Oct 31 04:03:51.695523 ip-10-0-140-110 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 04:03:52.118925 ip-10-0-140-110 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:03:52.118925 ip-10-0-140-110 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 04:03:52.118925 ip-10-0-140-110 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:03:52.118925 ip-10-0-140-110 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126192    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126270    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126274    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126277    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126281    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.126274 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126284    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126287    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126290    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126293    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126296    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126298    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126301    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126304    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126306    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126311    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126315    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126318    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126321    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126324    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126327    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126330    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126333    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126335    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126338    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.126499 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126341    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126351    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126354    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126356    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126359    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126362    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126365    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126367    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126370    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126372    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126375    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126377    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126380    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126383    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126386    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126389    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126392    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126394    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126397    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126400    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.126951 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126402    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126405    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126407    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126410    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126412    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126415    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126418    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126421    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126423    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126426    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126428    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126431    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126434    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126436    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126439    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126441    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126444    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126446    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126449    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.127482 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126451    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126457    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126460    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126463    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126466    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126468    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126471    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126474    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126477    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126480    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126482    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126485    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126487    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126491    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126495    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126498    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126501    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126504    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126507    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.127925 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126510    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126513    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126516    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126519    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126522    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126524    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126527    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126530    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126532    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126535    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126537    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126540    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126542    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126545    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126550    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126552    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126555    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126558    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126560    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126564    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.128403 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126568    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.128907 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126572    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.128907 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126575    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.128907 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.126577    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128960    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128968    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128971    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128974    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128979    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.128981 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128984    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128987    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128990    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128993    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128995    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.128999    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129001    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129004    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129007    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129009    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129012    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129014    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129017    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129020    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129022    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129025    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129028    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129030    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129033    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129035    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.129128 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129038    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129041    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129043    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129046    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129048    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129050    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129053    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129055    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129058    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129061    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129063    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129066    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129068    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129070    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129075    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129079    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129082    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129085    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129089    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.129630 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129091    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129094    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129096    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129099    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129101    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129104    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129106    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129109    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129111    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129115    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129117    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129120    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129122    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129125    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129127    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129130    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129133    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129135    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129138    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.130082 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129141    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129144    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129146    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129150    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129167    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129170    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129173    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129176    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129179    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129182    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129184    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129187    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129190    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129193    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129197    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129199    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129202    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129204    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129207    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129209    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.130564 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129212    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129214    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129217    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129219    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129222    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129224    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129227    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129229    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129232    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129234    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129237    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129240    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129242    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129245    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129248    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129251    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129254    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129256    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129259    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129261    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.131028 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129264    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129267    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129270    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129783    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129788    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129792    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129795    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129797    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129800    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129803    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129806    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129809    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129811    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129814    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129817    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129819    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129821    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129824    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129827    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129829    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.131519 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129832    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129834    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129837    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129840    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129843    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129845    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129848    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129860    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129863    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129867    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129869    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129872    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129875    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129877    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129880    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129883    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129887    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129891    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129895    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.131983 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129898    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129903    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129907    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129910    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129913    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129915    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129918    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129920    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129923    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129925    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129928    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129931    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129933    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129936    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129938    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129941    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129944    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129946    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.132459 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129949    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129954    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129956    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129959    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129962    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129964    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129968    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129970    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129973    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129975    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129978    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129981    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129983    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129985    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129989    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129992    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129994    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.129997    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130000    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130002    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.132898 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130005    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130007    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130010    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130012    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130015    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130017    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130020    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130022    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130025    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130027    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130030    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130032    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130035    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130039    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130042    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130044    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130047    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130049    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130052    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130055    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.133402 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130058    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130061    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130066    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130068    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130071    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130073    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130076    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130079    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130082    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130084    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.130087    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131187    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131197    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131206    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131212    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131217    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131220    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131224    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131228    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131231    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 04:03:52.133895 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131235    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131238    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131241    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131244    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131248    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131251    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131254    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131257    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131260    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131264    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131267    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131270    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131273    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131276    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131280    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131283    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131286    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131289    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131292    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131296    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131299    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131302    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131305    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131309    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 04:03:52.134395 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131312    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131317    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131320    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131323    2571 flags.go:64] FLAG: --enable-server="true"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131326    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131331    2571 flags.go:64] FLAG: --event-burst="100"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131334    2571 flags.go:64] FLAG: --event-qps="50"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131337    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131340    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131343    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131347    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131350    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131354    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131357    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131360    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131363    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131366    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131369    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131371    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131374    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131377    2571 flags.go:64] FLAG: --feature-gates=""
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131381    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131384    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131387    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131390    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131393    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 04:03:52.135028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131397    2571 flags.go:64] FLAG: --help="false"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131400    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131404    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131407    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131410    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131413    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131416    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131420    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131423    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131426    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131429    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131432    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131436    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131438    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131442    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131445    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131448    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131451    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131454    2571 flags.go:64] FLAG: --lock-file=""
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131457    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131460    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131462    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131468    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 04:03:52.135713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131471    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131474    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131477    2571 flags.go:64] FLAG: --logging-format="text"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131480    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131484    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131487    2571 flags.go:64] FLAG: --manifest-url=""
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131490    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131494    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131497    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131501    2571 flags.go:64] FLAG: --max-pods="110"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131504    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131507    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131510    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131513    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131516    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131519    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131525    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131532    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131535    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131539    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131542    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131545    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131549    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131552    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131555    2571 flags.go:64] FLAG: --port="10250"
Oct 31 04:03:52.136297 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131558    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131561    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008953caa12f115f7"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131564    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131567    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131570    2571 flags.go:64] FLAG: --register-node="true"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131573    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131577    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131579    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131582    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131585    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131589    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131592    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131596    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131598    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131601    2571 flags.go:64] FLAG: --runonce="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131604    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131607    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131610    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131613    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131616    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131624    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131628    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131631    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131635    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131638    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 04:03:52.136985 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131641    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131644    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131648    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131651    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131654    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131660    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131663    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131666    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131670    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131673    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131676    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131679    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131682    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131685    2571 flags.go:64] FLAG: --v="2"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131690    2571 flags.go:64] FLAG: --version="false"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131695    2571 flags.go:64] FLAG: --vmodule=""
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131699    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.131702    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131854    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131858    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131862    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131865    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131867    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131870    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.137617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131873    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131875    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131878    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131880    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131883    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131886    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131890    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131893    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131895    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131898    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131903    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131906    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131909    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131911    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131914    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131917    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131920    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131923    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131925    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131928    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.138262 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131930    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131933    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131936    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131939    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131941    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131944    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131947    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131949    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131952    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131954    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131957    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131961    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131964    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131967    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131970    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131972    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131975    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131978    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131982    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.138743 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131985    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131988    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131991    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131994    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.131997    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132000    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132002    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132005    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132008    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132011    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132014    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132016    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132019    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132021    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132024    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132027    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132029    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132032    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132035    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.139207 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132037    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132040    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132042    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132046    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132050    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132053    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132056    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132059    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132062    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132064    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132067    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132070    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132073    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132076    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132079    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132081    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132085    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132088    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132090    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.139655 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132093    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132096    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132098    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132101    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132103    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132106    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132109    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132111    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132114    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132117    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132119    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132122    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132125    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132127    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132130    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132133    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132135    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132138    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132140    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132143    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.140124 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132146    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132148    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132151    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.132194    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132233    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132238    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132242    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132245    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132247    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132250    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132254    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132257    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132260    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132263    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132265    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132268    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.140634 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132271    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132273    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132276    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132279    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132281    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132284    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132287    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132289    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132292    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132294    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132297    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132300    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132302    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132305    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132307    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132310    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132312    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132315    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132317    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.141022 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132320    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132324    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132328    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132331    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132334    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132337    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132340    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132343    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132346    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132349    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132352    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132354    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132358    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132360    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132363    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132365    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132368    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132370    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132373    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.141485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132376    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132378    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132381    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132383    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132386    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132389    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132391    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132394    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132396    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132399    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132401    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132404    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132408    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132411    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132415    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132418    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132421    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132424    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132427    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.141942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132429    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132433    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132436    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132438    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132441    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132443    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132446    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132449    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132451    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132453    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132456    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132459    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132461    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132464    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132467    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132469    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132472    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132475    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132481    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132486    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.142429 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132488    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132491    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132493    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132496    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132498    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132501    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132503    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132506    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132508    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132511    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132515    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132517    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132520    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132524    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132526    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132529    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.132531    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.139791    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.139805    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 04:03:52.142940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.140532    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.140539    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140612    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140618    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140622    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140625    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140628    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140631    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140634    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140637    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140640    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140644    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140646    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140649    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140652    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140655    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140657    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140660    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140663    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140665    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.143411 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140668    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140671    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140673    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140676    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140680    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140683    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140687    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140692    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140696    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140700    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140703    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140707    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140709    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140712    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140715    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140717    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140720    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140723    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140726    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.143874 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140729    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140731    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140734    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140737    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140739    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140742    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140745    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140747    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140750    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140753    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140755    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140758    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140761    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140763    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140766    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140769    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140771    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140774    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140776    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140779    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.144395 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140782    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140785    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140787    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140790    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140793    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140796    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140799    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140802    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140805    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140807    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140810    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140812    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140815    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140818    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140821    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140825    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140829    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140833    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140837    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.144867 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140840    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140844    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140848    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140852    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140855    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140859    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140863    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140867    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140871    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140873    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140876    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140878    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140881    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140884    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140887    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140890    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140893    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140897    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140899    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140902    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.145463 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140905    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140908    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140911    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140913    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140916    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140918    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140921    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140923    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140926    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.140928    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.140957    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141170    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141177    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141180    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141184    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.145942 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141187    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141189    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141192    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141195    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141197    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141200    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141203    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141205    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141208    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141211    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141213    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141217    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141220    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141222    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141225    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141228    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141231    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141234    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141236    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.146479 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141239    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141242    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141244    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141247    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141249    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141252    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141254    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141257    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141260    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141262    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141265    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141267    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141270    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141273    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141275    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141278    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141280    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141283    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141285    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141288    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.146958 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141290    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141293    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141295    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141298    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141300    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141305    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141309    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141312    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141315    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141319    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141323    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141326    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141329    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141331    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141334    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141347    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141350    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141353    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141356    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.147778 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141358    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141361    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141363    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141366    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141369    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141372    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141374    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141377    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141379    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141382    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141384    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141387    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141389    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141392    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141394    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141397    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141399    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141402    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141405    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.148485 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141413    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141416    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141419    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141422    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141425    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141427    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141430    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141432    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141435    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141437    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141440    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141442    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141445    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141447    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141450    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141452    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141455    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141457    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141460    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141462    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.149008 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141465    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141467    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141470    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141472    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141475    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.141498    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141542    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141547    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141550    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141553    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141557    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141560    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141563    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141576    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141579    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:03:52.149512 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141582    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141584    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141587    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141590    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141592    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141595    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141597    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141600    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141603    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141605    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141607    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141610    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141613    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141615    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141618    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141621    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141623    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141626    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141628    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:03:52.150041 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141631    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141633    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141636    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141638    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141641    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141643    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141646    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141648    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141651    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141654    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141656    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141659    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141667    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141670    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141672    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141675    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141677    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141680    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141682    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141685    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:03:52.150544 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141687    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141690    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141693    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141695    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141697    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141700    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141702    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141705    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141707    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141710    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141712    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141715    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141717    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141719    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141722    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141725    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141728    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141730    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141733    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:03:52.151130 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141736    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141738    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141741    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141743    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141746    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141748    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141761    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141764    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141766    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141769    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141771    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141774    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141776    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141779    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141781    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141784    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141787    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141789    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141791    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141794    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:03:52.151617 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141796    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141799    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141801    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141803    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141806    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141808    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141811    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141813    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141816    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141818    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141821    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141824    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141826    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141829    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141831    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141834    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141836    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141839    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:52.141842    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:03:52.152122 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.141911    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.144066    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.145053    2571 server.go:1014] "Starting client certificate rotation"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.145185    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.145243    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.147249    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 04:03:52.152577 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.148782    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 04:03:52.158660 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.158640    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 31 04:03:52.162032 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.162015    2571 log.go:25] "Validated CRI v1 image API"
Oct 31 04:03:52.162793 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.162769    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 04:03:52.165692 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.165665    2571 fs.go:135] Filesystem UUIDs: map[06da1b3c-7e10-4a4a-bd2c-7cdcbc520783:/dev/nvme0n1p4 4036d856-6e78-4628-9882-d04d37d31c5a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 04:03:52.165772 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.165692    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 04:03:52.171663 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.171541    2571 manager.go:217] Machine: {Timestamp:2026-10-31 04:03:52.170278345 +0000 UTC m=+0.350984631 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101765 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7202890d9c05b4e7c37cd61900e1 SystemUUID:ec2f7202-890d-9c05-b4e7-c37cd61900e1 BootID:ea258aa9-96ca-4867-a474-06e311dd9703 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:06:e3:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:06:e3:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:be:4a:4c:c0:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 04:03:52.171663 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.171656    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 04:03:52.171883 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.171819    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 04:03:52.174971 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.174937    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 04:03:52.175164 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.174974    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 04:03:52.175213 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.175166    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 04:03:52.175213 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.175175    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 04:03:52.175213 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.175207    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 04:03:52.175336 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.175226    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 04:03:52.176245 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.176230    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 04:03:52.179360 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.179342    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 04:03:52.179507 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.179497    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 04:03:52.180296 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180284    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 04:03:52.180344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180304    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 04:03:52.180344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180316    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 04:03:52.180344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180326    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 31 04:03:52.180344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180334    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 04:03:52.180735 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.180714    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 04:03:52.183444 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.183424    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 04:03:52.189532 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.189515    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 04:03:52.190487 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190473    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 04:03:52.190531 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190508    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 04:03:52.190673 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190663    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 04:03:52.190711 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190679    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 04:03:52.190711 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190686    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 04:03:52.190711 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190692    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 04:03:52.190711 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190698    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 04:03:52.190711 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.190704    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 04:03:52.191466 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191456    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 04:03:52.191499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191467    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 04:03:52.191499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191475    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 04:03:52.191499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191481    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 04:03:52.191499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191496    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 04:03:52.191650 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.191505    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 04:03:52.192072 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.192061    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 04:03:52.192072 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.192072    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 04:03:52.196113 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.196095    2571 server.go:1263] "Started kubelet"
Oct 31 04:03:52.196251 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.196212    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 04:03:52.196847 ip-10-0-140-110 systemd[1]: Started Kubernetes Kubelet.
Oct 31 04:03:52.197006 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.196842    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 04:03:52.197006 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.196970    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 04:03:52.198105 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.198083    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 04:03:52.200033 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.200014    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 04:03:52.203899 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.203847    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 04:03:52.204171 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.204139    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 04:03:52.204516 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.204498    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.204990 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.204975    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 04:03:52.205450 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.204976    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 04:03:52.205540 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.205464    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 04:03:52.206046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206027    2571 factory.go:55] Registering systemd factory
Oct 31 04:03:52.206146 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206132    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 04:03:52.206234 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206224    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 04:03:52.206307 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.206256    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 04:03:52.206467 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206446    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 31 04:03:52.206822 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206806    2571 factory.go:153] Registering CRI-O factory
Oct 31 04:03:52.206822 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206824    2571 factory.go:223] Registration of the crio container factory successfully
Oct 31 04:03:52.206944 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206896    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 04:03:52.206944 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.206917    2571 factory.go:103] Registering Raw factory
Oct 31 04:03:52.207030 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.207006    2571 manager.go:1199] Started watching for new ooms in manager
Oct 31 04:03:52.208007 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.207990    2571 manager.go:319] Starting recovery of all containers
Oct 31 04:03:52.219478 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.219456    2571 manager.go:324] Recovery completed
Oct 31 04:03:52.223712 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.223697    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.225491 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.225473    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.225549 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.225510    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.225549 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.225524    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.226048 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226030    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 31 04:03:52.226048 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226044    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 04:03:52.226196 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226065    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 04:03:52.226196 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226080    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.226891 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226872    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.226964 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226908    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.226964 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.226923    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.227288 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.227275    2571 policy_none.go:50] "Start"
Oct 31 04:03:52.227348 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.227290    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 04:03:52.227348 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.227302    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 04:03:52.228269 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.228257    2571 policy_none.go:44] "Start"
Oct 31 04:03:52.235057 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.235031    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 04:03:52.235141 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.235057    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 04:03:52.236092 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.235033    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18e38178bd54281e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-10-31 04:03:52.196057118 +0000 UTC m=+0.376763403,LastTimestamp:2026-10-31 04:03:52.196057118 +0000 UTC m=+0.376763403,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}"
Oct 31 04:03:52.248684 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.248529    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 04:03:52.250485 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.250234    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 04:03:52.250485 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.250244    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 04:03:52.250485 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.250383    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s6h7h"
Oct 31 04:03:52.261819 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.261782    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 04:03:52.261992 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.261972    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s6h7h"
Oct 31 04:03:52.266119 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266105    2571 manager.go:341] "Starting Device Plugin manager"
Oct 31 04:03:52.266219 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.266132    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 04:03:52.266219 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266141    2571 server.go:87] "Starting device plugin registration server"
Oct 31 04:03:52.266359 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266348    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 04:03:52.266402 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266359    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 04:03:52.266734 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266518    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 04:03:52.266734 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266592    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 04:03:52.266734 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.266599    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 04:03:52.267077 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.267055    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 04:03:52.267172 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.267104    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.285399 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.285378    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:52.337559 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.337519    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 04:03:52.338778 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.338761    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 04:03:52.338827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.338788    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 04:03:52.338827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.338813    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 04:03:52.338887 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.338852    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 04:03:52.356787 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.356765    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:52.366696 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.366680    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.367539 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.367515    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.367620 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.367553    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.367620 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.367571    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.367620 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.367595    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.388170 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.388096    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.388170 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.388127    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-110.ec2.internal\": node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.439519 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.439472    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"]
Oct 31 04:03:52.439677 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.439550    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.440699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.440683    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.440757 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.440713    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.440757 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.440723    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.442063 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.442051    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.442207 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.442194    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.442252 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.442222    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.443069 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443046    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.443069 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443060    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.443069 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443075    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.443301 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443089    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.443301 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443089    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.443301 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.443178    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.444413 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.444396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.444481 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.444423    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:52.445141 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.445125    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:52.445226 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.445165    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:52.445226 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.445179    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:52.466233 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.466213    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.469956 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.469934    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.473660 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.473641    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.507800 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.507766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.507800 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.507797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.507962 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.507818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.566985 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.566956    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.608522 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.608522 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.608708 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.608708 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608596    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.608708 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.608708 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.608608    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.668076 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.667991    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.769475 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.769439    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.773894 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.773874    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.777292 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:52.777260    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:52.874330 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.874295    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:52.978831 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:52.978762    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.078869 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.078847    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.103696 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.103672    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:53.145060 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.145020    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 04:03:53.145623 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.145243    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 04:03:53.145623 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.145251    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 04:03:53.145623 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.145252    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 04:03:53.180472 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.180425    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.263314 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.263244    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:03:53.265060 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.265039    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:03:53.265172 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.265069    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:03:53.265172 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.265080    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:03:53.266888 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.266858    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:52 +0000 UTC" deadline="2028-07-02 14:15:54.415084117 +0000 UTC"
Oct 31 04:03:53.266888 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.266884    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14650h12m1.148203526s"
Oct 31 04:03:53.281112 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.281091    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.327070 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:53.327032    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd74d1416e17adb1d88ce1b2d585255.slice/crio-dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718 WatchSource:0}: Error finding container dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718: Status 404 returned error can't find the container with id dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718
Oct 31 04:03:53.333566 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.333546    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:03:53.342086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.342041    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerStarted","Data":"dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718"}
Oct 31 04:03:53.375999 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:53.375965    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50080b7deadb8e30dee000b7f0bbadfa.slice/crio-45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c WatchSource:0}: Error finding container 45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c: Status 404 returned error can't find the container with id 45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c
Oct 31 04:03:53.382210 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.382186    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.485626 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.485579    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.586339 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.586247    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.686963 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:53.686915    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:03:53.775235 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.775206    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:53.804694 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.804656    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:53.830701 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.830668    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:03:53.832157 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.832139    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:03:53.856675 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:53.856602    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:03:54.181026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.180944    2571 apiserver.go:52] "Watching apiserver"
Oct 31 04:03:54.205742 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.205703    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 04:03:54.227855 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.227820    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 04:03:54.228659 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.228221    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-8rscm","openshift-network-diagnostics/network-check-target-w49lj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c","openshift-multus/network-metrics-daemon-dk458","kube-system/konnectivity-agent-gk4cs","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal","openshift-image-registry/node-ca-xvz9v","openshift-ovn-kubernetes/ovnkube-node-vg88m","kube-system/global-pull-secret-syncer-s4d8g","openshift-cluster-node-tuning-operator/tuned-rsdcv","openshift-multus/multus-kgjcj","openshift-network-operator/iptables-alerter-tqzjj","openshift-dns/node-resolver-phzfq"]
Oct 31 04:03:54.230102 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.230074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.231261 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.231240    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.233114 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.232856    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:54.233114 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.232929    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:03:54.234383 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.234362    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.235839 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.235727    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.235839 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.235802    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:03:54.237260 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.237226    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.238899 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.238876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.238989 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.238942    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:03:54.241278 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.241170    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.243092 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.243001    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.244064 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.244044    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.246969 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.246949    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.247286 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.247263    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 04:03:54.247460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.247444    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 04:03:54.247601 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.247578    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\""
Oct 31 04:03:54.247749 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.247733    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.247820 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.247776    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 04:03:54.248038 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.248004    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 04:03:54.248950 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.248930    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.270785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.270748    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 04:03:54.270889 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.270828    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-htk6t"
Oct 31 04:03:54.306864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.306839    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 04:03:54.320284 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992wh\" (UniqueName: \"kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320424    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320489    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320577    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sf9t\" (UniqueName: \"kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kmp\" (UniqueName: \"kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.320869 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320820    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.320989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321016    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.321462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321295    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321369    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4z92\" (UniqueName: \"kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt257\" (UniqueName: \"kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321479    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322026 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321669    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5w5\" (UniqueName: \"kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsm8k\" (UniqueName: \"kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.321959    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.322715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322049    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322200    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8k98\" (UniqueName: \"kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.323396 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.322253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.349352 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.349308    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"50080b7deadb8e30dee000b7f0bbadfa","Type":"ContainerStarted","Data":"45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c"}
Oct 31 04:03:54.356975 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.356946    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.357127 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357063    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.357214 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357127    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 04:03:54.357214 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357142    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.357214 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357178    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 04:03:54.357388 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357288    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.357388 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357329    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 04:03:54.357388 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357362    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357580    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357608    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357651    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357698    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357718    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 04:03:54.357818 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.357794    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360059    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360104    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360263    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360307    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360328    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360381    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360407    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360430    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360451    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360493    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360505    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360532    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 04:03:54.360569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.360454    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 04:03:54.371312 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.371288    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\""
Oct 31 04:03:54.371312 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.371293    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 04:03:54.371852 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.371826    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\""
Oct 31 04:03:54.376127 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.376109    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:54.422986 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.422950    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.422996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423098    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4z92\" (UniqueName: \"kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423162    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.423199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423193    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt257\" (UniqueName: \"kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423178    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423225    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423233    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423237    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423454    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.423537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423523    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423548    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423599    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423613    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423643    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423694    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423725    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423827    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423842    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424001 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5w5\" (UniqueName: \"kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsm8k\" (UniqueName: \"kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423950    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.423962    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424003    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424134    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424212    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424224    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424279    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424323    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.424900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424370    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424374    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424413    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8k98\" (UniqueName: \"kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424482    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424619    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992wh\" (UniqueName: \"kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.425857 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424766    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424779    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424824    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424881    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424742    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424929    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.424969    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425025    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425188    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425266    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425285    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425312    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.426727 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425323    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf9t\" (UniqueName: \"kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425389    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425433    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kmp\" (UniqueName: \"kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425538    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425579    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425584    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425640    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425658    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.427621 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.425905    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426062    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426158    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426251    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426276    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426428    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.428495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426670    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426802    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426873    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426927    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426931    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.426980    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.427051    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.427160    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.927136133 +0000 UTC m=+3.107842434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427344    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.429389 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427418    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427476    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.427494    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427509    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427542    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.427562    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.927544302 +0000 UTC m=+3.108250595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427571    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427627    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427681    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427800    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.427892    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.428103    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.428551    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.428878    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.429360    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.429942    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.430224 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.429945    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.430827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.430420    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.451518 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.451303    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4z92\" (UniqueName: \"kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.451518 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.451356    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt257\" (UniqueName: \"kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.452072 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.451945    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsm8k\" (UniqueName: \"kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.452202 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.452069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kmp\" (UniqueName: \"kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.453644 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.453622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8k98\" (UniqueName: \"kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.454236 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.454204    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5w5\" (UniqueName: \"kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.454661 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.454636    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992wh\" (UniqueName: \"kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.454871 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.454848    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:03:54.454950 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.454875    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:03:54.454950 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.454903    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:54.455068 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.454969    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:54.9549483 +0000 UTC m=+3.135654600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:54.455193 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.455157    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.456973 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.456924    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf9t\" (UniqueName: \"kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.559049 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.559007    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:03:54.561072 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.561023    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:03:54.566920 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.566825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kgjcj"
Oct 31 04:03:54.572838 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.572805    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca197fb2_de4b_4936_9af4_e1177179830c.slice/crio-68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477 WatchSource:0}: Error finding container 68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477: Status 404 returned error can't find the container with id 68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477
Oct 31 04:03:54.574240 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.574213    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:03:54.575407 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.575349    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57f88b0_c977_4b88_82b3_14f7b7792845.slice/crio-030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a WatchSource:0}: Error finding container 030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a: Status 404 returned error can't find the container with id 030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a
Oct 31 04:03:54.581819 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.581792    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a1ddf5_4fd2_4b73_bf05_9e15e10aba57.slice/crio-ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129 WatchSource:0}: Error finding container ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129: Status 404 returned error can't find the container with id ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129
Oct 31 04:03:54.582481 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.582458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:03:54.585471 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.585393    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ce4d22_fc05_4428_a9dd_651a3b1c4686.slice/crio-18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7 WatchSource:0}: Error finding container 18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7: Status 404 returned error can't find the container with id 18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7
Oct 31 04:03:54.589327 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.589304    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:03:54.593265 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.593219    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7768dc_fc1a_405d_92cc_75e17eaccf19.slice/crio-d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f WatchSource:0}: Error finding container d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f: Status 404 returned error can't find the container with id d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f
Oct 31 04:03:54.604414 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.604380    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:03:54.615254 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.615203    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce70e1ef_553c_437b_bb60_f046bf05fa2e.slice/crio-bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897 WatchSource:0}: Error finding container bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897: Status 404 returned error can't find the container with id bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897
Oct 31 04:03:54.618815 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.618771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:03:54.624602 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.624419    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:03:54.633591 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.633553    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e10b9e_7427_4d6d_9a99_e17824c49078.slice/crio-e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420 WatchSource:0}: Error finding container e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420: Status 404 returned error can't find the container with id e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420
Oct 31 04:03:54.637112 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:03:54.637080    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf269a0_e3d7_4932_ba07_f4f48ee0c787.slice/crio-de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8 WatchSource:0}: Error finding container de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8: Status 404 returned error can't find the container with id de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8
Oct 31 04:03:54.710892 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.710813    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:54.930699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.930664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:54.930699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:54.930710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:54.930917 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.930870    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:54.930953 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.930936    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:55.930916638 +0000 UTC m=+4.111622916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:54.931123 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.931100    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:54.931231 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:54.931172    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:55.931154825 +0000 UTC m=+4.111861114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:55.032568 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.032248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:55.032568 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.032406    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:03:55.032568 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.032545    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:03:55.032568 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.032560    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:55.032851 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.032636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:56.032612619 +0000 UTC m=+4.213318912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:55.273803 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.273594    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:54 +0000 UTC" deadline="2028-07-03 11:28:34.638752124 +0000 UTC"
Oct 31 04:03:55.273803 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.273661    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14671h24m39.365095473s"
Oct 31 04:03:55.367590 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.367428    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerStarted","Data":"e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420"}
Oct 31 04:03:55.372840 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.372746    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c" event={"ID":"ce70e1ef-553c-437b-bb60-f046bf05fa2e","Type":"ContainerStarted","Data":"bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897"}
Oct 31 04:03:55.381199 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.381039    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsdcv" event={"ID":"5460fbe1-498a-4aae-b32d-b8d97666d7b0","Type":"ContainerStarted","Data":"63dd4b597aafaf531829e3e4e6ddfbcd49334d906eb83da23a7af214ddecb6c2"}
Oct 31 04:03:55.382939 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.382852    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvz9v" event={"ID":"ca7768dc-fc1a-405d-92cc-75e17eaccf19","Type":"ContainerStarted","Data":"d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f"}
Oct 31 04:03:55.392361 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.392141    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477"}
Oct 31 04:03:55.405900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.405692    2571 generic.go:356] "Generic (PLEG): container finished" podID="3cd74d1416e17adb1d88ce1b2d585255" containerID="715644b5e2af6b0c1d7bef4e1cf556a4d7911e41368fcbe56916d02f9d75a05c" exitCode=0
Oct 31 04:03:55.405900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.405814    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerDied","Data":"715644b5e2af6b0c1d7bef4e1cf556a4d7911e41368fcbe56916d02f9d75a05c"}
Oct 31 04:03:55.415967 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.414990    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"50080b7deadb8e30dee000b7f0bbadfa","Type":"ContainerStarted","Data":"75eda65653178c60d3f20f2189774dff8f7fdaf637f27dc72292ed0d1a220e8d"}
Oct 31 04:03:55.422576 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.422399    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-phzfq" event={"ID":"acf269a0-e3d7-4932-ba07-f4f48ee0c787","Type":"ContainerStarted","Data":"de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8"}
Oct 31 04:03:55.430102 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.429996    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gk4cs" event={"ID":"95ce4d22-fc05-4428-a9dd-651a3b1c4686","Type":"ContainerStarted","Data":"18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7"}
Oct 31 04:03:55.444142 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.444065    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgjcj" event={"ID":"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57","Type":"ContainerStarted","Data":"ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129"}
Oct 31 04:03:55.487270 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.487130    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqzjj" event={"ID":"e57f88b0-c977-4b88-82b3-14f7b7792845","Type":"ContainerStarted","Data":"030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a"}
Oct 31 04:03:55.498707 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.498492    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" podStartSLOduration=1.4984620180000001 podStartE2EDuration="1.498462018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:55.497282413 +0000 UTC m=+3.677988713" watchObservedRunningTime="2026-10-31 04:03:55.498462018 +0000 UTC m=+3.679168318"
Oct 31 04:03:55.745828 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.745679    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.940779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:55.940836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.941023    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.941099    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.941071707 +0000 UTC m=+6.121777981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.941534    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:55.941675 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:55.941585    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:57.941569354 +0000 UTC m=+6.122275633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:56.042585 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.041841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:56.042585 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.042047    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:03:56.042585 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.042074    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:03:56.042585 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.042089    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:56.042585 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.042155    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:03:58.042132351 +0000 UTC m=+6.222838628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:56.274864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.274766    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:54 +0000 UTC" deadline="2028-07-03 11:23:39.375637126 +0000 UTC"
Oct 31 04:03:56.274864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.274811    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14671h19m43.100830942s"
Oct 31 04:03:56.345505 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.345428    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:56.345717 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.345626    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:03:56.346005 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.345809    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:56.346005 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.345428    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:56.346152 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.346132    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:03:56.346237 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:56.345965    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:03:56.502049 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:56.501903    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerStarted","Data":"6de74c6023bb68d8b9b5aa5520eca8ed10a4930b6f09d2a851fd78ecd59f744e"}
Oct 31 04:03:57.961724 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:57.961630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:57.961724 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:57.961690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:57.962303 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:57.961857    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:57.962303 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:57.961930    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:04:01.961907148 +0000 UTC m=+10.142613438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:03:57.962420 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:57.962386    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:57.962479 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:57.962440    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:01.962424481 +0000 UTC m=+10.143130772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:03:58.063800 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.063111    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:58.063800 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.063310    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:03:58.063800 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.063335    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:03:58.063800 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.063349    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:58.063800 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.063411    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:02.063392812 +0000 UTC m=+10.244099092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:03:58.341930 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.341893    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:03:58.342261 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.342107    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:03:58.342261 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.342191    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:03:58.342415 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.342304    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:03:58.342415 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.342354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:03:58.342532 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:03:58.342424    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:03:58.520500 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.520015    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gk4cs" event={"ID":"95ce4d22-fc05-4428-a9dd-651a3b1c4686","Type":"ContainerStarted","Data":"689f7750a2496705bd4a119af3df1c2a951e7791e3ff8442f44100d072f478a6"}
Oct 31 04:03:58.524133 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.524054    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="f55a3aed5f02f12405ffceefca58d38d9b56267be40fde8e51693ba1667e6fcd" exitCode=0
Oct 31 04:03:58.524133 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.524104    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"f55a3aed5f02f12405ffceefca58d38d9b56267be40fde8e51693ba1667e6fcd"}
Oct 31 04:03:58.566483 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.566416    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" podStartSLOduration=4.566394215 podStartE2EDuration="4.566394215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:56.548342899 +0000 UTC m=+4.729049202" watchObservedRunningTime="2026-10-31 04:03:58.566394215 +0000 UTC m=+6.747100515"
Oct 31 04:03:58.566864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.566826    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gk4cs" podStartSLOduration=2.882269207 podStartE2EDuration="6.566811024s" totalImagesPullingTime="3.684541817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:58.565460712 +0000 UTC m=+6.746167014" watchObservedRunningTime="2026-10-31 04:03:58.566811024 +0000 UTC m=+6.747517327"
Oct 31 04:03:58.675628 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.675576    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:58.905315 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:03:58.905218    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:00.296378 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.296251    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:00.341940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.341276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:00.341940 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:00.341398    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:00.341940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.341772    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:00.341940 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:00.341848    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:00.342237 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.341971    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:00.342237 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:00.342043    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:00.513275 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.513236    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:00.532239 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.532200    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c" event={"ID":"ce70e1ef-553c-437b-bb60-f046bf05fa2e","Type":"ContainerStarted","Data":"7bd414544e01cb41344c92126b98e37388dbd6b549dfb756d71956b24cb5ebbd"}
Oct 31 04:04:00.537090 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.537051    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvz9v" event={"ID":"ca7768dc-fc1a-405d-92cc-75e17eaccf19","Type":"ContainerStarted","Data":"64da077c5b28feccc23b63df22ffde56b1637d8bbdb9ff151ffab0c1acb2a0b5"}
Oct 31 04:04:00.729703 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.729569    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:00.940445 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:00.940406    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:01.151069 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:01.150958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:01.993242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:01.993291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:01.993464    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:01.993529    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.993511213 +0000 UTC m=+18.174217509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:01.993943    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:01.994030 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:01.993995    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:09.993979439 +0000 UTC m=+18.174685730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:02.021944 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.021886    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 04:04:02.068505 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.068467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:02.094049 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.093969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:02.094223 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.094171    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:04:02.094223 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.094196    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:04:02.094223 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.094209    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:02.094398 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.094264    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:10.094246597 +0000 UTC m=+18.274952884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:02.271246 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.270573    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:04:02.021907516Z","UUID":"b104dcb4-312c-4926-973e-f25795bbbc3d","Handler":null,"Name":"","Endpoint":""}
Oct 31 04:04:02.273335 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.272868    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 04:04:02.273335 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.272897    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.339940    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.340064    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.340443    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.340536    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.341681    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:02.342485 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:02.341766    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:02.441419 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.441330    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:02.554831 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.554794    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c" event={"ID":"ce70e1ef-553c-437b-bb60-f046bf05fa2e","Type":"ContainerStarted","Data":"c891734172d4154d6888cf3646aaf1f393643f2f635c4c64114975640f74ed56"}
Oct 31 04:04:02.558891 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.558041    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kgjcj" event={"ID":"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57","Type":"ContainerStarted","Data":"405dd0b104858a0d896866c4f493dfd0e61a5bbed52d57961c0b45fa0fed6f2b"}
Oct 31 04:04:02.604436 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.603558    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kgjcj" podStartSLOduration=2.877150778 podStartE2EDuration="10.603542316s" totalImagesPullingTime="7.726391538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:02.603205184 +0000 UTC m=+10.783911509" watchObservedRunningTime="2026-10-31 04:04:02.603542316 +0000 UTC m=+10.784248612"
Oct 31 04:04:02.604436 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.604172    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xvz9v" podStartSLOduration=4.597846025 podStartE2EDuration="9.604146351s" totalImagesPullingTime="5.006300326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:00.586226371 +0000 UTC m=+8.766932667" watchObservedRunningTime="2026-10-31 04:04:02.604146351 +0000 UTC m=+10.784852648"
Oct 31 04:04:02.657304 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.657227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:02.875950 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:02.875674    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:03.090920 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.090876    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:03.271015 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.270793    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:04:03.271858 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.271693    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:04:03.297974 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.297941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:03.561623 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.561542    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-phzfq" event={"ID":"acf269a0-e3d7-4932-ba07-f4f48ee0c787","Type":"ContainerStarted","Data":"2919ea355ba4ee34910499e1e10726adb63263a5861cb4761fa651ebcddd716a"}
Oct 31 04:04:03.563936 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.563905    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="e6e3fa0fdc7106d91e9665146f2667ed30cfc9a689052bb76e369632bfc208a6" exitCode=0
Oct 31 04:04:03.564052 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.563996    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"e6e3fa0fdc7106d91e9665146f2667ed30cfc9a689052bb76e369632bfc208a6"}
Oct 31 04:04:03.567588 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.567561    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c" event={"ID":"ce70e1ef-553c-437b-bb60-f046bf05fa2e","Type":"ContainerStarted","Data":"11249befc9c147a2523365f62c76c3fa1e39ad5c352c398b5b812ad00886c008"}
Oct 31 04:04:03.603873 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.603812    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-phzfq" podStartSLOduration=2.45345898 podStartE2EDuration="10.603792852s" totalImagesPullingTime="8.150333872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:03.603564546 +0000 UTC m=+11.784270854" watchObservedRunningTime="2026-10-31 04:04:03.603792852 +0000 UTC m=+11.784499148"
Oct 31 04:04:03.650767 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.650717    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c" podStartSLOduration=2.609504678 podStartE2EDuration="10.650702766s" totalImagesPullingTime="8.041198088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:03.650201713 +0000 UTC m=+11.830908007" watchObservedRunningTime="2026-10-31 04:04:03.650702766 +0000 UTC m=+11.831409060"
Oct 31 04:04:03.677394 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:03.677347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:04.270507 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.270472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.340146    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.340146    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:04.340274    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.340279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:04.340361    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:04.340600 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:04.340444    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:04.525946 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.525906    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:04.570866 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.570834    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqzjj" event={"ID":"e57f88b0-c977-4b88-82b3-14f7b7792845","Type":"ContainerStarted","Data":"7b4de23bae1c9be32daadee3ee969c27c01177c8b6e85a19ef71679590de3f9b"}
Oct 31 04:04:04.612889 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.612775    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tqzjj" podStartSLOduration=4.664030781 podStartE2EDuration="12.61275616s" totalImagesPullingTime="7.948725379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:04.611852015 +0000 UTC m=+12.792558327" watchObservedRunningTime="2026-10-31 04:04:04.61275616 +0000 UTC m=+12.793462458"
Oct 31 04:04:04.870078 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:04.870036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:05.084167 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.084121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:05.436887 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.436800    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:05.576061 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.576030    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="d11c1218053f905ce03a48f7896e10a97cf595ea560de2a98d6a749893bbcf4a" exitCode=0
Oct 31 04:04:05.576249 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.576128    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"d11c1218053f905ce03a48f7896e10a97cf595ea560de2a98d6a749893bbcf4a"}
Oct 31 04:04:05.578030 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.578007    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rsdcv" event={"ID":"5460fbe1-498a-4aae-b32d-b8d97666d7b0","Type":"ContainerStarted","Data":"dc59ec4443c2c6a594808c73075bfe96535340f8aa9f846d648df857079a72d6"}
Oct 31 04:04:05.622572 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.622544    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-phzfq_acf269a0-e3d7-4932-ba07-f4f48ee0c787/dns-node-resolver/0.log"
Oct 31 04:04:05.676863 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.676818    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:05.700382 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:05.700276    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rsdcv" podStartSLOduration=2.252617004 podStartE2EDuration="12.700259683s" totalImagesPullingTime="10.447642679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:05.699766634 +0000 UTC m=+13.880472928" watchObservedRunningTime="2026-10-31 04:04:05.700259683 +0000 UTC m=+13.880965977"
Oct 31 04:04:06.266746 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.266712    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:06.339478 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.339439    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:06.339638 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.339453    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:06.339638 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:06.339573    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:06.339638 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.339593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:06.339810 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:06.339714    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:06.339861 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:06.339809    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:06.473894 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.473866    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:06.582843 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.582761    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="cf33b53f1641bb69002728bca642b5d4d858b166cdc141f93683515fb6ba36f9" exitCode=0
Oct 31 04:04:06.582996 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.582851    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"cf33b53f1641bb69002728bca642b5d4d858b166cdc141f93683515fb6ba36f9"}
Oct 31 04:04:06.624310 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.624278    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvz9v_ca7768dc-fc1a-405d-92cc-75e17eaccf19/node-ca/0.log"
Oct 31 04:04:06.872176 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:06.871898    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:07.091043 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:07.091014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:07.306064 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:07.305983    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:07.673141 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:07.673068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:07.886656 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:07.886615    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:08.274284 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.274247    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:08.339482 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.339258    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:08.339482 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.339258    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:08.339482 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:08.339393    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:08.339482 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.339428    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:08.339761 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:08.339532    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:08.339761 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:08.339591    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:08.488439 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.488389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:08.590779 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.590743    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="ef262139e74a5bfa8af9b13e2d8b7edeece050ce1aa26a82858c64b997c53ce2" exitCode=0
Oct 31 04:04:08.590945 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.590788    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"ef262139e74a5bfa8af9b13e2d8b7edeece050ce1aa26a82858c64b997c53ce2"}
Oct 31 04:04:08.875703 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:08.875628    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:09.088342 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.088313    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:09.596094 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.595862    2571 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="e89fdd7f791029b713ad7754dfe2b6238c12f34881644f2b2fdfdcd1034b46f5" exitCode=0
Oct 31 04:04:09.596094 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.595944    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerDied","Data":"e89fdd7f791029b713ad7754dfe2b6238c12f34881644f2b2fdfdcd1034b46f5"}
Oct 31 04:04:09.599317 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599299    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"1e9fc1e2325b0f95705e7c20eb8b60ae1fcdef196f1b724e289ddfa14c353e67"}
Oct 31 04:04:09.599386 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599353    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"5471a202858d911db597378c7f4511e2fc8c0200a4b13c7f1909190640a06966"}
Oct 31 04:04:09.599386 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599384    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"532db938c75c719bdaec83de1377d834b3a0070695c97f980dbbedc340ff5fdf"}
Oct 31 04:04:09.599460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599412    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"342676b3d11a0a85ca0773ae8014a122c90821d9102b80157242fa1d732187fd"}
Oct 31 04:04:09.599460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599438    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"36a8ac6cf6883903cc5d1c60640f5c4917a5aac91a8871a7cb5379471f05bc30"}
Oct 31 04:04:09.599515 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.599463    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"e9414f72d50c33ff6c4ffca366f502a3afc2629a1121fcb5e9e644ebdc42b996"}
Oct 31 04:04:09.670758 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:09.670715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:10.057804 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.057723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:10.057804 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.057764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:10.058353 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.057873    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:10.058353 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.057881    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:10.058353 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.057933    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:04:26.057917543 +0000 UTC m=+34.238623831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:10.058353 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.057952    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:26.057943509 +0000 UTC m=+34.238649796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:10.158111 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.158064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:10.158354 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.158192    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:04:10.158354 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.158212    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:04:10.158354 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.158222    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:10.158354 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.158278    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:26.158260795 +0000 UTC m=+34.338967082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:10.270146 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.270101    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:10.339504 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.339422    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:10.339504 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.339487    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:10.339504 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.339500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:10.339720 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.339609    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:10.339720 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.339645    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:10.339720 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:10.339698    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:10.477767 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.477735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:10.604856 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.604777    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rscm" event={"ID":"b6e10b9e-7427-4d6d-9a99-e17824c49078","Type":"ContainerStarted","Data":"5c5127ec722606ed98e001ded162db1b21a6c3a44641865c08259c931ac66949"}
Oct 31 04:04:10.652053 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.651993    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8rscm" podStartSLOduration=6.755532821 podStartE2EDuration="17.651974894s" totalImagesPullingTime="10.896442073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:10.651674478 +0000 UTC m=+18.832380775" watchObservedRunningTime="2026-10-31 04:04:10.651974894 +0000 UTC m=+18.832681190"
Oct 31 04:04:10.901191 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:10.901102    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:11.118283 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:11.118249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:11.324610 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:11.324534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:11.609537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:11.609453    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"55e626a0cd3422a08f9ba0dcce264f750f0433f662ae9ed0b0fe2cb862994a03"}
Oct 31 04:04:11.676581 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:11.676552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:12.272084 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.272054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:12.339706 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.339674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:12.339848 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:12.339754    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:12.339848 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.339770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:12.339848 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.339796    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:12.339954 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:12.339861    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:12.339954 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:12.339917    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:12.490531 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.490501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:12.872653 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:12.872619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:13.086940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.086910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:13.501294 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.501249    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:13.617377 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.617342    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" event={"ID":"ca197fb2-de4b-4936-9af4-e1177179830c","Type":"ContainerStarted","Data":"5e630cb98cd12ae2e9c6e13b7035bf5e5d067643458c84d44c11aa484ea67fb5"}
Oct 31 04:04:13.617624 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.617606    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:13.617694 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.617682    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:13.617745 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.617733    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:13.633250 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.633226    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:13.633388 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.633377    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:13.713648 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.713532    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:13.714302 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:13.714268    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m" podStartSLOduration=7.315331496 podStartE2EDuration="21.714255669s" totalImagesPullingTime="14.398924173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:13.700504062 +0000 UTC m=+21.881210357" watchObservedRunningTime="2026-10-31 04:04:13.714255669 +0000 UTC m=+21.894961964"
Oct 31 04:04:14.168383 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:14.168303    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:14.339336 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:14.339295    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:14.339336 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:14.339331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:14.339545 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:14.339304    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:14.339545 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:14.339443    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:14.339545 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:14.339507    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:14.339545 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:14.339535    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:16.339850 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:16.339816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:16.339850 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:16.339849    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:16.340368 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:16.339816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:16.340368 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:16.339927    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:16.340368 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:16.339992    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:16.340368 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:16.340051    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:18.339646 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:18.339612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:18.340200 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:18.339620    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:18.340200 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:18.339716    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:18.340200 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:18.339761    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:18.340200 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:18.339841    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:18.340200 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:18.339890    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:20.021254 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.021214    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:04:20.021872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.021338    2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 04:04:20.021872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.021843    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:04:20.339133 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.339096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:20.339340 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.339109    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:20.339340 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:20.339239    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:20.339340 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.339267    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:20.339340 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:20.339316    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:20.339556 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:20.339379    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:20.349809 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.349783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:20.565809 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.565781    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:20.776695 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.776618    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:20.985957 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:20.985920    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:21.191237 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:21.191209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:22.147440 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.147407    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:22.340495 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.340465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:22.340904 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:22.340594    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:22.340904 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.340634    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:22.340904 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.340688    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:22.340904 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:22.340756    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:22.340904 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:22.340826    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:22.354796 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.354770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:22.559009 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:22.558975    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:23.553453 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.553426    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4d8g"]
Oct 31 04:04:23.553885 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.553523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:23.553885 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:23.553604    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:23.557079 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.557056    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w49lj"]
Oct 31 04:04:23.557215 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.557103    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dk458"]
Oct 31 04:04:23.557215 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.557176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:23.557335 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:23.557253    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:23.557335 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:23.557290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:23.557335 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:23.557330    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:25.340080 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:25.340049    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:25.340080 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:25.340071    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:25.340548 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:25.340197    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:25.340548 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:25.340233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:25.340548 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:25.340329    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:25.340548 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:25.340391    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:25.410512 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:25.410484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:25.621657 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:25.621583    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:26.066774 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:26.066747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:26.066922 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:26.066779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:26.066922 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.066900    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:26.066998 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.066916    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:26.066998 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.066958    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret podName:4502cc34-9b28-4f7e-a13b-594221372bbf nodeName:}" failed. No retries permitted until 2026-10-31 04:04:58.066942162 +0000 UTC m=+66.247648437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret") pod "global-pull-secret-syncer-s4d8g" (UID: "4502cc34-9b28-4f7e-a13b-594221372bbf") : object "kube-system"/"original-pull-secret" not registered
Oct 31 04:04:26.066998 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.066979    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs podName:88c15711-14f6-494e-a518-478f433cfeb1 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:58.066968558 +0000 UTC m=+66.247674831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs") pod "network-metrics-daemon-dk458" (UID: "88c15711-14f6-494e-a518-478f433cfeb1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 04:04:26.167702 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:26.167668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:26.167863 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.167790    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 04:04:26.167863 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.167805    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 04:04:26.167863 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.167815    2571 projected.go:196] Error preparing data for projected volume kube-api-access-c5lg9 for pod openshift-network-diagnostics/network-check-target-w49lj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:26.167968 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:26.167865    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9 podName:ac673d9c-d013-487f-828d-72c57b8386b7 nodeName:}" failed. No retries permitted until 2026-10-31 04:04:58.167850372 +0000 UTC m=+66.348556661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5lg9" (UniqueName: "kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9") pod "network-check-target-w49lj" (UID: "ac673d9c-d013-487f-828d-72c57b8386b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 04:04:27.339661 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:27.339479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:27.340038 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:27.339524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:27.340038 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:27.339736    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s4d8g" podUID="4502cc34-9b28-4f7e-a13b-594221372bbf"
Oct 31 04:04:27.340038 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:27.339529    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:27.340038 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:27.339810    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w49lj" podUID="ac673d9c-d013-487f-828d-72c57b8386b7"
Oct 31 04:04:27.340038 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:27.339882    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dk458" podUID="88c15711-14f6-494e-a518-478f433cfeb1"
Oct 31 04:04:27.795828 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:27.795799    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-phzfq_acf269a0-e3d7-4932-ba07-f4f48ee0c787/dns-node-resolver/0.log"
Oct 31 04:04:28.133596 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.133571    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeReady"
Oct 31 04:04:28.133731 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.133684    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 04:04:28.194483 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.194437    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jphhc"]
Oct 31 04:04:28.216663 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.216628    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gqp52"]
Oct 31 04:04:28.216820 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.216786    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.232355 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.232322    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v6nhz"]
Oct 31 04:04:28.232475 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.232434    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.232858 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.232823    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 04:04:28.233028 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.233005    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 04:04:28.233124 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.233012    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 04:04:28.233124 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.233069    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w2pm\""
Oct 31 04:04:28.233124 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.233076    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 04:04:28.246384 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.246363    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jphhc"]
Oct 31 04:04:28.246492 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.246401    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqp52"]
Oct 31 04:04:28.246492 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.246415    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6nhz"]
Oct 31 04:04:28.246600 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.246500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.284344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.284322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.284448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.284347    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.284448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.284366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt4q\" (UniqueName: \"kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.284448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.284431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.284565 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.284550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.332847 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332817    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 04:04:28.332847 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332832    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 04:04:28.333051 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332912    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gw6dk\""
Oct 31 04:04:28.333051 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332936    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzkqw\""
Oct 31 04:04:28.333051 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332957    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 04:04:28.333051 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332961    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 04:04:28.333051 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.332968    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 04:04:28.385368 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385287    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385368 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385347    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385398    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt4q\" (UniqueName: \"kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nc9\" (UniqueName: \"kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385700    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385786    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.385860 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385805    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrs52\" (UniqueName: \"kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.386198 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.385969    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.389321 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.389299    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.410045 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.410019    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt4q\" (UniqueName: \"kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.486122 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.486122 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.486342 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs52\" (UniqueName: \"kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.486342 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486226    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.486342 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nc9\" (UniqueName: \"kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.486342 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.486560 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486538    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.486841 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.486815    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.488640 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.488618    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.488763 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.488742    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.510715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.510686    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nc9\" (UniqueName: \"kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.510804 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.510687    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs52\" (UniqueName: \"kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.545868 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.545839    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:28.561361 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.561340    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:04:28.571478 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.571455    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:04:28.587706 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.587678    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:28.752902 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.752855    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:28.968626 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.968544    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqp52"]
Oct 31 04:04:28.972007 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.971976    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v6nhz"]
Oct 31 04:04:28.972353 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:28.972331    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83f2509_20b7_4ece_9477_abeda7a82df1.slice/crio-3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0 WatchSource:0}: Error finding container 3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0: Status 404 returned error can't find the container with id 3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0
Oct 31 04:04:28.981579 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.981554    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jphhc"]
Oct 31 04:04:28.986185 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:28.986151    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79174e7_f9ee_4451_a10f_c8ffb4b4f948.slice/crio-dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3 WatchSource:0}: Error finding container dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3: Status 404 returned error can't find the container with id dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3
Oct 31 04:04:28.986754 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:28.986732    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d7df10_6133_426c_a8e1_48c78cdc1f44.slice/crio-9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764 WatchSource:0}: Error finding container 9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764: Status 404 returned error can't find the container with id 9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764
Oct 31 04:04:28.995505 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:28.995484    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvz9v_ca7768dc-fc1a-405d-92cc-75e17eaccf19/node-ca/0.log"
Oct 31 04:04:29.339674 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.339480    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:29.339846 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.339508    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:29.339846 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.339551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:29.356303 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356278    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\""
Oct 31 04:04:29.356438 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356309    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 04:04:29.356438 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356384    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 04:04:29.356438 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356390    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 04:04:29.356562 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356458    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\""
Oct 31 04:04:29.356611 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356559    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\""
Oct 31 04:04:29.356721 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.356696    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 04:04:29.656262 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.656181    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jphhc" event={"ID":"f79174e7-f9ee-4451-a10f-c8ffb4b4f948","Type":"ContainerStarted","Data":"d0eb3aa48b295f1a7fbdc26ed5cad857e3a67b4e19e209d2a7cc058a23381871"}
Oct 31 04:04:29.657098 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.656277    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jphhc" event={"ID":"f79174e7-f9ee-4451-a10f-c8ffb4b4f948","Type":"ContainerStarted","Data":"dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3"}
Oct 31 04:04:29.657548 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.657518    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6nhz" event={"ID":"b5d7df10-6133-426c-a8e1-48c78cdc1f44","Type":"ContainerStarted","Data":"9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764"}
Oct 31 04:04:29.658964 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:29.658940    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqp52" event={"ID":"c83f2509-20b7-4ece-9477-abeda7a82df1","Type":"ContainerStarted","Data":"3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0"}
Oct 31 04:04:30.664266 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:30.664226    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jphhc" event={"ID":"f79174e7-f9ee-4451-a10f-c8ffb4b4f948","Type":"ContainerStarted","Data":"8ce5733c5b2a806eae5f61acc5dfb5e7cc323fd9dc4d2254ada3c94f0f96db13"}
Oct 31 04:04:30.861481 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:30.861436    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:31.005649 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.005560    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"]
Oct 31 04:04:31.018210 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.018181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.031909 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.031880    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"]
Oct 31 04:04:31.044475 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.044447    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 04:04:31.044598 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.044540    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 04:04:31.044661 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.044610    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 04:04:31.047506 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.047438    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 04:04:31.047506 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.047478    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tnrrl\""
Oct 31 04:04:31.047506 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.047495    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 04:04:31.059572 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.059222    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"]
Oct 31 04:04:31.074603 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.074566    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qfnsk"]
Oct 31 04:04:31.074746 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.074720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.089011 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.088983    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"]
Oct 31 04:04:31.089187 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.089129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.089655 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.089623    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:31.105813 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.105784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.105988 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.105831    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.105988 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.105872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.105988 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.105925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wdt\" (UniqueName: \"kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.128045 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.128018    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 04:04:31.128391 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.128359    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 04:04:31.128466 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.128449    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 04:04:31.128555 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.128536    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 04:04:31.128718 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.128703    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5nqr4\""
Oct 31 04:04:31.134837 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.134817    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 04:04:31.135215 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.134844    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 04:04:31.135215 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.134848    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfl64\""
Oct 31 04:04:31.207166 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.207302 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207302 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207302 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.207302 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.207503 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.207503 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbshb\" (UniqueName: \"kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.207503 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.207503 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207701 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207814 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207871 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjvx\" (UniqueName: \"kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.207927 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.207873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.208065 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.208129 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.208129 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.208291 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wdt\" (UniqueName: \"kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.208350 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.208482 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.208469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.212840 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.212688    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.217772 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.217746    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.220831 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.220798    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.237340 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.237309    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wdt\" (UniqueName: \"kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.311477 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311592 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311592 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311586    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311699 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbshb\" (UniqueName: \"kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.311827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjvx\" (UniqueName: \"kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.311940 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.312049 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.311988    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.312135 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.312117    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.312904 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.312881    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.312972 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.312953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.313241 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.313221    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.313499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.313480    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.316979 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.316526    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.316979 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.316568    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.316979 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.316859    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.317606 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.317581    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.318562 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.318538    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.318836 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.318811    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.320123 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.320102    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.338176 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.338136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:31.341046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.341022    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjvx\" (UniqueName: \"kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.341500 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.341367    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbshb\" (UniqueName: \"kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.390035 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.389795    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:04:31.432465 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.432271    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:04:31.490557 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.490524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:04:31.552881 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.552847    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:31.669702 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.669672    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6nhz" event={"ID":"b5d7df10-6133-426c-a8e1-48c78cdc1f44","Type":"ContainerStarted","Data":"47e22ca2c6666c6b273f43b1c7a0ed0948907a9bdc7eb1c9262737c7b84e8332"}
Oct 31 04:04:31.670192 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.669770    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v6nhz" event={"ID":"b5d7df10-6133-426c-a8e1-48c78cdc1f44","Type":"ContainerStarted","Data":"9c1d26ab3f2fffabe8841aef7e9fd8febbb649404c3b46bdf5da0e829ac2a0bb"}
Oct 31 04:04:31.670590 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.670566    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:31.672020 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.671982    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfnsk" event={"ID":"f60643a3-3f4e-4308-a52e-0893b0fce7eb","Type":"ContainerStarted","Data":"6a979666eb6613c6097b776e7288cf2f9b4603eb00bbde9291dddcf00596c814"}
Oct 31 04:04:31.714639 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.714590    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-v6nhz" podStartSLOduration=1.6825721420000002 podStartE2EDuration="3.714570797s" totalImagesPullingTime="2.031998655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:31.713794451 +0000 UTC m=+39.894500745" watchObservedRunningTime="2026-10-31 04:04:31.714570797 +0000 UTC m=+39.895277096"
Oct 31 04:04:31.743406 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.743231    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"]
Oct 31 04:04:31.747070 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:31.747040    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda345e1d5_f60a_473a_9cef_11db043e79df.slice/crio-43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095 WatchSource:0}: Error finding container 43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095: Status 404 returned error can't find the container with id 43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095
Oct 31 04:04:31.760897 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.760869    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:31.773959 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.773588    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"]
Oct 31 04:04:31.779013 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:31.778866    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8cf693_3022_4ad6_a50a_82466ad009ba.slice/crio-588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc WatchSource:0}: Error finding container 588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc: Status 404 returned error can't find the container with id 588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc
Oct 31 04:04:31.970960 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:31.970723    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.277886 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.277800    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.479179 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.479143    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:04:32.507825 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.507799    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.511255 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.511231    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:04:32.520937 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.520908    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.524777 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.524599    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 04:04:32.524777 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.524736    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 04:04:32.524777 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.524760    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 04:04:32.525018 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.524805    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 04:04:32.525186 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.525111    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 04:04:32.526194 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.525474    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 04:04:32.526194 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.525543    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 04:04:32.526194 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.525831    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-87lk2\""
Oct 31 04:04:32.538792 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.538770    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 04:04:32.539076 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.539059    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 04:04:32.621733 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.621908 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.621908 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.621908 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.621908 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.621908 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621923    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.621981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.622026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.622041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.622130 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.622068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.678584 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.678544    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d" event={"ID":"a345e1d5-f60a-473a-9cef-11db043e79df","Type":"ContainerStarted","Data":"c30fa14bf22a6c4006205d0b8a9752c3a87032a1541f4cc9815944c9f5a98fb7"}
Oct 31 04:04:32.679462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.678625    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d" event={"ID":"a345e1d5-f60a-473a-9cef-11db043e79df","Type":"ContainerStarted","Data":"58b218d6905cc93eeb140713719c118a271680e40aed1c11a1e11adb372a0b70"}
Oct 31 04:04:32.679462 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.678654    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d" event={"ID":"a345e1d5-f60a-473a-9cef-11db043e79df","Type":"ContainerStarted","Data":"43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095"}
Oct 31 04:04:32.680639 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.680615    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqp52" event={"ID":"c83f2509-20b7-4ece-9477-abeda7a82df1","Type":"ContainerStarted","Data":"666d6dfdc0552d0d0e5080eae3c30487313a73c87b80a0da9a8cd98c45482f69"}
Oct 31 04:04:32.682418 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.682392    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4" event={"ID":"de8cf693-3022-4ad6-a50a-82466ad009ba","Type":"ContainerStarted","Data":"588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc"}
Oct 31 04:04:32.685503 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.685478    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jphhc" event={"ID":"f79174e7-f9ee-4451-a10f-c8ffb4b4f948","Type":"ContainerStarted","Data":"b7f2e68199ed2258280ba7c88dd9b42cd4812c407228afc705c1de5482972fee"}
Oct 31 04:04:32.722532 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722532 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.722785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723086 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.722944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.723332 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.723216    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.725127 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.725097    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.725406 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.725381    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.726007 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.725900    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gqp52" podStartSLOduration=1.219114705 podStartE2EDuration="4.725885272s" totalImagesPullingTime="3.506770567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:32.725554286 +0000 UTC m=+40.906260599" watchObservedRunningTime="2026-10-31 04:04:32.725885272 +0000 UTC m=+40.906591568"
Oct 31 04:04:32.731548 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.731501    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.731790 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.731716    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.733496 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.733473    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.733875 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.733825    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.734114 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.734055    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.734408 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.734392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.738069 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.738044    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.738958 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.738914    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.740978 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.740875    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.747952 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.747902    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:32.770706 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.770656    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jphhc" podStartSLOduration=1.488206703 podStartE2EDuration="4.770642561s" totalImagesPullingTime="3.282435858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:04:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:32.769385917 +0000 UTC m=+40.950092238" watchObservedRunningTime="2026-10-31 04:04:32.770642561 +0000 UTC m=+40.951348856"
Oct 31 04:04:32.774592 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.774489    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.959537 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:32.959319    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:04:33.270023 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.269993    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"]
Oct 31 04:04:33.291493 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.291460    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.306509 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.303287    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"]
Oct 31 04:04:33.307747 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307716    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 04:04:33.307864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307755    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ttr22\""
Oct 31 04:04:33.307864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307794    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 04:04:33.307864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307719    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 04:04:33.307864 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307858    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-76bmm1hqvtd1j\""
Oct 31 04:04:33.308036 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307876    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 04:04:33.308036 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.307944    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 04:04:33.322374 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.322341    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:04:33.362799 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:33.362759    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337b4d1b_8c2b_4ba1_a463_38484614334c.slice/crio-ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5 WatchSource:0}: Error finding container ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5: Status 404 returned error can't find the container with id ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5
Oct 31 04:04:33.429987 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.429961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430081 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.429999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430191 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430191 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx49j\" (UniqueName: \"kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430283 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430329 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430381 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430381 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.430514 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.430482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.531910 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.531873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532073 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.531943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532073 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.531968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532073 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.531988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532073 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.532019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx49j\" (UniqueName: \"kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532073 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.532064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532351 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.532100    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532351 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.532134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.532351 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.532171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.533330 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.533268    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.533446 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.533406    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.535949 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.535920    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.536046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.535928    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.536046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.536001    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.536132 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.536041    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.536132 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.536085    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.536557 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.536533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.556725 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.556696    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx49j\" (UniqueName: \"kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.615620 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.615589    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:33.690493 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.690463    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5"}
Oct 31 04:04:33.691979 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.691950    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfnsk" event={"ID":"f60643a3-3f4e-4308-a52e-0893b0fce7eb","Type":"ContainerStarted","Data":"5bf300c98e417c02fc5e652fda8f6cea54bba355898d988e5e42666c27999528"}
Oct 31 04:04:33.693657 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.693639    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:33.837899 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:33.837748    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:34.300710 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.300387    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"]
Oct 31 04:04:34.305690 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:34.305656    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f518cc_282f_4dc6_9279_4e162eba47ab.slice/crio-9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91 WatchSource:0}: Error finding container 9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91: Status 404 returned error can't find the container with id 9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91
Oct 31 04:04:34.426212 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.426124    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:34.643475 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.643407    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:34.698760 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.698670    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d" event={"ID":"a345e1d5-f60a-473a-9cef-11db043e79df","Type":"ContainerStarted","Data":"c256a039735fedca59735bd3076053c644a1f0660f29b9ca94fe8daf6a3f356b"}
Oct 31 04:04:34.700582 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.700556    2571 generic.go:356] "Generic (PLEG): container finished" podID="f60643a3-3f4e-4308-a52e-0893b0fce7eb" containerID="5bf300c98e417c02fc5e652fda8f6cea54bba355898d988e5e42666c27999528" exitCode=0
Oct 31 04:04:34.700698 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.700649    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfnsk" event={"ID":"f60643a3-3f4e-4308-a52e-0893b0fce7eb","Type":"ContainerDied","Data":"5bf300c98e417c02fc5e652fda8f6cea54bba355898d988e5e42666c27999528"}
Oct 31 04:04:34.703075 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.703050    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4" event={"ID":"de8cf693-3022-4ad6-a50a-82466ad009ba","Type":"ContainerStarted","Data":"9a868e13957fda13970fb89edc997ac8cab76aa4eb85e9e0b62f3ba611287ef9"}
Oct 31 04:04:34.703312 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.703291    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4" event={"ID":"de8cf693-3022-4ad6-a50a-82466ad009ba","Type":"ContainerStarted","Data":"d6ffba77b5444e7a40dfff297c83b4de69e6b9e318990af48d4bd634002f1f5c"}
Oct 31 04:04:34.703368 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.703336    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4" event={"ID":"de8cf693-3022-4ad6-a50a-82466ad009ba","Type":"ContainerStarted","Data":"06c74c1bfcc27513d37434169b1b2f011bac1a4bfc36938d4d9807a4d56cf64b"}
Oct 31 04:04:34.704477 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.704455    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91"}
Oct 31 04:04:34.746478 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.744794    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d" podStartSLOduration=2.754488567 podStartE2EDuration="4.744775647s" totalImagesPullingTime="1.99028708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:34.742539218 +0000 UTC m=+42.923245513" watchObservedRunningTime="2026-10-31 04:04:34.744775647 +0000 UTC m=+42.925481943"
Oct 31 04:04:34.856532 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:34.856479    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4" podStartSLOduration=1.6714786670000001 podStartE2EDuration="3.85646509s" totalImagesPullingTime="2.184986423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:34.855617682 +0000 UTC m=+43.036323977" watchObservedRunningTime="2026-10-31 04:04:34.85646509 +0000 UTC m=+43.037171384"
Oct 31 04:04:35.371496 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.371462    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.584034 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.584003    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.665050 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.665020    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"]
Oct 31 04:04:35.667373 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.667348    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.696461 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.696430    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 04:04:35.696773 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.696753    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-27zhj\""
Oct 31 04:04:35.696852 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.696806    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 04:04:35.696852 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.696758    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 04:04:35.696946 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.696896    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 04:04:35.697106 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.697084    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bebll3gva5m81\""
Oct 31 04:04:35.698367 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.698340    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"]
Oct 31 04:04:35.709900 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.709875    2571 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="eb0af5a40286d72459810c7bd19b4357a25fc11b71d96369fbabfd6433e26ed1" exitCode=0
Oct 31 04:04:35.710586 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.709964    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"eb0af5a40286d72459810c7bd19b4357a25fc11b71d96369fbabfd6433e26ed1"}
Oct 31 04:04:35.712486 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.712463    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfnsk" event={"ID":"f60643a3-3f4e-4308-a52e-0893b0fce7eb","Type":"ContainerStarted","Data":"3f7a4ade4d0b3b7f23ec50c1c54ee342384cec345883d83a9cfe4b93dae7e2a0"}
Oct 31 04:04:35.712571 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.712521    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfnsk" event={"ID":"f60643a3-3f4e-4308-a52e-0893b0fce7eb","Type":"ContainerStarted","Data":"6091675e273d6807b039e2cf186a2308ce0043955aa3b67a161de2adf666a6ff"}
Oct 31 04:04:35.757452 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757452 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757455    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmff\" (UniqueName: \"kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757622 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757622 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757700 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757743 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757707    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.757743 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.757723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.814430 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.814376    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qfnsk" podStartSLOduration=2.8156750600000002 podStartE2EDuration="4.814354758s" totalImagesPullingTime="1.884079383s" totalInitContainerRuntime="114.600315ms" isStatefulPod=true podCreationTimestamp="2026-10-31 04:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:35.813203526 +0000 UTC m=+43.993909821" watchObservedRunningTime="2026-10-31 04:04:35.814354758 +0000 UTC m=+43.995061053"
Oct 31 04:04:35.858814 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.858735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.858977 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.858816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmff\" (UniqueName: \"kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.858977 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.858836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.859338 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.859309    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.859432 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.859389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.859630 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.859602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.859747 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.859641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.860097 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.860058    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.860330 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.860320    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.861292 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.861183    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.862327 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.862306    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.862448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.862305    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.862513 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.862492    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.883392 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.883364    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmff\" (UniqueName: \"kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:35.894235 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.894207    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"]
Oct 31 04:04:35.897071 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.897051    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:35.915534 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.915472    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 04:04:35.915534 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.915523    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5b7hl\""
Oct 31 04:04:35.916658 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.916635    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"]
Oct 31 04:04:35.953363 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.953324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.978698 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:35.978668    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:36.061671 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.061642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.061823 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.061780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht87\" (UniqueName: \"kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.163245 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.163119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.163245 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.163216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht87\" (UniqueName: \"kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.166519 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.166496    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.168369 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.168339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:36.187646 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.187620    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht87\" (UniqueName: \"kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.208315 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.208283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:36.291572 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.291537    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"]
Oct 31 04:04:36.297509 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:36.297468    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6c8de0_72bd_40d8_aa86_34805cf518d0.slice/crio-1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db WatchSource:0}: Error finding container 1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db: Status 404 returned error can't find the container with id 1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db
Oct 31 04:04:36.355811 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.355770    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"]
Oct 31 04:04:36.361102 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.361076    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.392316 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392286    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:36.392612 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392595    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 04:04:36.392687 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392634    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 04:04:36.392753 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392737    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ks42h\""
Oct 31 04:04:36.392827 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392810    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 04:04:36.392876 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392859    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 04:04:36.392949 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.392931    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 04:04:36.393024 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.393011    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 04:04:36.395347 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.395324    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"]
Oct 31 04:04:36.466486 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466486 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466474    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466640 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466640 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466640 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466785 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466821 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbvn\" (UniqueName: \"kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.466850 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.466830    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.518960 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.518923    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"]
Oct 31 04:04:36.523480 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:36.523447    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbb956a_7849_497a_9463_f85599ef4d57.slice/crio-48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db WatchSource:0}: Error finding container 48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db: Status 404 returned error can't find the container with id 48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db
Oct 31 04:04:36.567686 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.567686 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbvn\" (UniqueName: \"kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.567912 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.567912 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.567912 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.567912 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.568122 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.568122 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.567959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.569046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.568319    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.569046 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.569005    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.569380 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.569356    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.571003 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.570978    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.571191 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.571141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.571318 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.571298    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.572333 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.572306    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.592347 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.592317    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbvn\" (UniqueName: \"kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.614377 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.614333    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:36.679443 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.679413    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:04:36.718666 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.718530    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf" event={"ID":"1fbb956a-7849-497a-9463-f85599ef4d57","Type":"ContainerStarted","Data":"48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db"}
Oct 31 04:04:36.720963 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.720901    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56" event={"ID":"eb6c8de0-72bd-40d8-aa86-34805cf518d0","Type":"ContainerStarted","Data":"1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db"}
Oct 31 04:04:36.897822 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.897789    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:04:36.900575 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.900552    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:36.916569 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916528    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tsb5s\""
Oct 31 04:04:36.916718 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916572    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 04:04:36.916718 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916575    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 04:04:36.916934 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916888    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 04:04:36.916934 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916912    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 04:04:36.916934 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916922    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 04:04:36.917147 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916950    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 04:04:36.917147 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.916972    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 04:04:36.931032 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.930985    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 04:04:36.932214 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:36.932186    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:04:37.011677 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.011539    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"]
Oct 31 04:04:37.019978 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:37.019946    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f8fddd_031d_48c8_a22c_4b5c0bf69c5a.slice/crio-d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026 WatchSource:0}: Error finding container d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026: Status 404 returned error can't find the container with id d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026
Oct 31 04:04:37.073914 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.073876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074075 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.073928    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074123 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074189 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074270 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074322 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.074448 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.074354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.176190 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.175472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.177896 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.177871    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.178589 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.178560    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.180198 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.179883    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.180686 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.180626    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.180962 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.180910    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.183696 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.183656    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.185056 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.184991    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.202504 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.202467    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.213351 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.213061    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:37.539341 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.539296    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:04:37.547507 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:37.547269    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074b81bc_3e3f_4fc3_b6af_8ade5ea6673d.slice/crio-1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153 WatchSource:0}: Error finding container 1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153: Status 404 returned error can't find the container with id 1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153
Oct 31 04:04:37.730290 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.730255    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"fe591afd8b50af1ef8d139bb12faca1e40859a84452fb240f0fb8ad04d3b7ba4"}
Oct 31 04:04:37.732262 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.731435    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"2a5a86341d6f7f635ee214342de3fa7618de35b6a5b2e37f619773513fb98b84"}
Oct 31 04:04:37.732440 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.732422    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"a60c0fe931c88d360beb0ef4a64b76b067d6aee8a38c8aaaeaaac3db5a8a2eb7"}
Oct 31 04:04:37.733067 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.733053    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8594bc5d5f-sn56k" event={"ID":"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d","Type":"ContainerStarted","Data":"1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153"}
Oct 31 04:04:37.735112 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.735036    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm" event={"ID":"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a","Type":"ContainerStarted","Data":"d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026"}
Oct 31 04:04:37.738430 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.738321    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"4f0a53c703af8e5bedcf012c45e85e9f67290e96efc17724250ce645533e4f25"}
Oct 31 04:04:37.738430 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.738382    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"285d8109a3ac4b8d5a9369971a58f7972d71c9cc5862ca4350835cc91c7081f9"}
Oct 31 04:04:37.998879 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:37.998830    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:04:38.002944 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.002405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.023562 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.023518    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 04:04:38.024030 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024004    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 04:04:38.024204 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024076    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 04:04:38.024204 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024143    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 04:04:38.024329 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024232    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 04:04:38.024329 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024271    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 04:04:38.024329 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024318    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 04:04:38.024460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024355    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 04:04:38.024460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024401    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 04:04:38.024460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024441    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 04:04:38.024675 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024654    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 04:04:38.024824 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.024811    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lw8lb\""
Oct 31 04:04:38.025024 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.025003    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-98quaske5a7de\""
Oct 31 04:04:38.025132 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.025096    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 04:04:38.033404 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.033379    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 04:04:38.033609 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.033585    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 04:04:38.049061 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.048979    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:04:38.084328 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084293    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084328 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084556 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084577    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.084951 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.084948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.085409 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.085055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.085409 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.085177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.102880 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.102844    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:38.185690 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185832    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.185872 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185926    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.185967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186125 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186298    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186464 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.186783 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.186762    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.187458 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.187436    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.191454 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.191430    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.192006 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.191990    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.192918 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.192652    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.193994 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.193674    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.195045 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.194724    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.195045 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.194996    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.198501 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.198450    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.199705 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.199678    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.200612 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.200226    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.200612 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.200280    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.200612 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.200575    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.201866 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.201848    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.202147 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.202133    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.208386 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.208329    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.208578 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.208558    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.208740 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.208718    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.213177 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.213106    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.232489 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.232420    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.328198 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.328149    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:38.731390 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.731313    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:04:38.762930 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:38.762888    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb56708_d5b3_47bc_9bb6_1d931829bc34.slice/crio-cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b WatchSource:0}: Error finding container cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b: Status 404 returned error can't find the container with id cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b
Oct 31 04:04:38.763121 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.763068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:38.767419 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.767262    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"5dce22cace3775452903aab027722ae61094dadea523e6371016b7871438707c"}
Oct 31 04:04:38.767419 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.767343    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"15a28ac42fa5b0e9dea504db790a499f8ab8576749caf0dc77f1346d991cb6aa"}
Oct 31 04:04:38.767419 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.767368    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"8afcd2d7211725b74f44efe040584c3b39449208d0c2ef1f6e98101551db0628"}
Oct 31 04:04:38.770480 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.769703    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56" event={"ID":"eb6c8de0-72bd-40d8-aa86-34805cf518d0","Type":"ContainerStarted","Data":"bcc49373d916539de1657080c061ddf19ae1ec0a697a6a9bc29d5ee1ad432b31"}
Oct 31 04:04:38.972283 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:38.972258    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:39.775816 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.775732    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm" event={"ID":"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a","Type":"ContainerStarted","Data":"a6c4408db99d98c0547757cf151331235972c8f963e71baaba1e2790f9ae1055"}
Oct 31 04:04:39.775816 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.775799    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm" event={"ID":"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a","Type":"ContainerStarted","Data":"d806a477442a0eeb555add21448949f214aa61073e04a1dc0d1a4dce788479c0"}
Oct 31 04:04:39.776293 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.775826    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm" event={"ID":"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a","Type":"ContainerStarted","Data":"78a66912c9dc8ab9258836accfa688d3f99aa6ff7393093649b34bde07ff7c46"}
Oct 31 04:04:39.779111 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.779082    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerStarted","Data":"1ac0225044e6ff6de1bfff7985284a23742883b88d4b966714ac08b1765e47fd"}
Oct 31 04:04:39.782004 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.781981    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"1560f207a0b6015b2d24b078935fd2a2c62a8e7e060a34981fb836492ce5eeed"}
Oct 31 04:04:39.782096 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.782047    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"c5384c26306c1f50da1ca21fc95a09100b70595e7778a7840f383e37f5c2cd74"}
Oct 31 04:04:39.782096 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.782090    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" event={"ID":"35f518cc-282f-4dc6-9279-4e162eba47ab","Type":"ContainerStarted","Data":"027bf948cc7e95c318598564e38f7b1dca7ab243d5a5484d6835c8d17db102b0"}
Oct 31 04:04:39.782200 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.782136    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:39.784094 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.784064    2571 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b" exitCode=0
Oct 31 04:04:39.784197 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.784141    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"}
Oct 31 04:04:39.784252 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.784227    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b"}
Oct 31 04:04:39.786102 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.786081    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf" event={"ID":"1fbb956a-7849-497a-9463-f85599ef4d57","Type":"ContainerStarted","Data":"426330ba4e1c680f52c70bc5e01e369b0c21a8c774a7dae2740f2ec299732aeb"}
Oct 31 04:04:39.825108 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.825060    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm" podStartSLOduration=1.406067913 podStartE2EDuration="3.825046816s" totalImagesPullingTime="2.418978903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:39.823363253 +0000 UTC m=+48.004069562" watchObservedRunningTime="2026-10-31 04:04:39.825046816 +0000 UTC m=+48.005753108"
Oct 31 04:04:39.825453 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.825421    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56" podStartSLOduration=2.843866559 podStartE2EDuration="4.825411629s" totalImagesPullingTime="1.98154507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:38.818080334 +0000 UTC m=+46.998786680" watchObservedRunningTime="2026-10-31 04:04:39.825411629 +0000 UTC m=+48.006117923"
Oct 31 04:04:39.866335 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.866272    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf" podStartSLOduration=2.482043654 podStartE2EDuration="4.866256463s" totalImagesPullingTime="2.384212809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:39.86474018 +0000 UTC m=+48.045446502" watchObservedRunningTime="2026-10-31 04:04:39.866256463 +0000 UTC m=+48.046962755"
Oct 31 04:04:39.978053 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:39.977993    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.423845653 podStartE2EDuration="7.977975067s" totalImagesPullingTime="4.537837844s" totalInitContainerRuntime="16.29157ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:39.976631192 +0000 UTC m=+48.157337488" watchObservedRunningTime="2026-10-31 04:04:39.977975067 +0000 UTC m=+48.158681365"
Oct 31 04:04:40.028145 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:40.028041    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw" podStartSLOduration=2.733026195 podStartE2EDuration="7.028021315s" totalImagesPullingTime="4.29499512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:40.025675617 +0000 UTC m=+48.206381961" watchObservedRunningTime="2026-10-31 04:04:40.028021315 +0000 UTC m=+48.208727611"
Oct 31 04:04:40.792995 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:40.792061    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:40.797595 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:40.797569    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:04:41.184183 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:41.184087    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:41.398430 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:41.398389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:41.797660 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:41.797622    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8594bc5d5f-sn56k" event={"ID":"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d","Type":"ContainerStarted","Data":"5169afc028c4bfe4de6e4f062faeb5a4d9d4858d30a6c7c6cf9f02d2633d8482"}
Oct 31 04:04:41.843461 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:41.843398    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8594bc5d5f-sn56k" podStartSLOduration=2.472001554 podStartE2EDuration="5.843379369s" totalImagesPullingTime="3.371377815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:41.842674174 +0000 UTC m=+50.023380470" watchObservedRunningTime="2026-10-31 04:04:41.843379369 +0000 UTC m=+50.024085667"
Oct 31 04:04:42.694421 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:42.694386    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:04:43.679945 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.679227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:43.810526 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810486    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"}
Oct 31 04:04:43.810715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810562    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"}
Oct 31 04:04:43.810715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810605    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"}
Oct 31 04:04:43.810715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810643    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"}
Oct 31 04:04:43.810715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810678    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"}
Oct 31 04:04:43.810715 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.810712    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerStarted","Data":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"}
Oct 31 04:04:43.889501 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.889448    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.738046893 podStartE2EDuration="6.88943379s" totalImagesPullingTime="3.13192323s" totalInitContainerRuntime="19.463667ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:43.888697035 +0000 UTC m=+52.069403330" watchObservedRunningTime="2026-10-31 04:04:43.88943379 +0000 UTC m=+52.070140085"
Oct 31 04:04:43.893713 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:43.893682    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:45.633150 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:45.633123    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:04:45.800885 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:45.800852    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:04:45.868807 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:45.868762    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:46.077313 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:46.077280    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:46.290600 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:46.290570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:46.308428 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:46.308395    2571 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:04:46.504484 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:46.504383    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:47.214633 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:47.214596    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:04:48.329433 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:48.329396    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:04:48.584005 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:48.583927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:48.796811 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:48.796769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:50.596055 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:50.596017    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:50.803056 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:50.803022    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:54.400989 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:54.400957    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:54.635612 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:54.635577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:55.979476 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:55.979442    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:55.979899 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:55.979501    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:04:58.085096 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.085055    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:58.085515 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.085129    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:58.100367 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.100339    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 04:04:58.107460 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.107438    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:58.113691 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.113672    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 04:04:58.117648 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.117630    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:58.185636 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.185604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:58.188778 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.188761    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\""
Oct 31 04:04:58.194825 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.194806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:04:58.200457 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.200437    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 04:04:58.221432 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.221389    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 04:04:58.224523 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.224503    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\""
Oct 31 04:04:58.229830 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.229799    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:58.230440 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.230423    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:04:58.495344 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.495275    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s4d8g"]
Oct 31 04:04:58.498214 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:58.498188    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4502cc34_9b28_4f7e_a13b_594221372bbf.slice/crio-c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5 WatchSource:0}: Error finding container c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5: Status 404 returned error can't find the container with id c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5
Oct 31 04:04:58.541182 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.541128    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\""
Oct 31 04:04:58.542197 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.542176    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dk458"]
Oct 31 04:04:58.543249 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.543235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:04:58.545318 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:58.545291    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c15711_14f6_494e_a518_478f433cfeb1.slice/crio-32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9 WatchSource:0}: Error finding container 32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9: Status 404 returned error can't find the container with id 32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9
Oct 31 04:04:58.591768 ip-10-0-140-110 kubenswrapper[2571]: E1031 04:04:58.590723    2571 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:04:58.680918 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.680881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:58.840854 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.840824    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w49lj"]
Oct 31 04:04:58.843762 ip-10-0-140-110 kubenswrapper[2571]: W1031 04:04:58.843735    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac673d9c_d013_487f_828d_72c57b8386b7.slice/crio-436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d WatchSource:0}: Error finding container 436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d: Status 404 returned error can't find the container with id 436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d
Oct 31 04:04:58.883200 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.883150    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w49lj" event={"ID":"ac673d9c-d013-487f-828d-72c57b8386b7","Type":"ContainerStarted","Data":"436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d"}
Oct 31 04:04:58.884394 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.884365    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dk458" event={"ID":"88c15711-14f6-494e-a518-478f433cfeb1","Type":"ContainerStarted","Data":"32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9"}
Oct 31 04:04:58.885499 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.885477    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s4d8g" event={"ID":"4502cc34-9b28-4f7e-a13b-594221372bbf","Type":"ContainerStarted","Data":"c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5"}
Oct 31 04:04:58.896041 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:58.896013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:59.193693 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:59.193572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:59.892998 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:59.892421    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dk458" event={"ID":"88c15711-14f6-494e-a518-478f433cfeb1","Type":"ContainerStarted","Data":"acbec04ce86e3b2cdf1549a9a55809cba7845699b8a1804dd6255b2e88f48d18"}
Oct 31 04:04:59.892998 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:59.892489    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dk458" event={"ID":"88c15711-14f6-494e-a518-478f433cfeb1","Type":"ContainerStarted","Data":"015b14f69e482255f0aa0edb7135c8ce0a8b570eed21e7b0667026c2eb5f539f"}
Oct 31 04:04:59.935100 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:04:59.935039    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dk458" podStartSLOduration=66.896818687 podStartE2EDuration="1m7.935027568s" totalImagesPullingTime="1.038208881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:59.93439879 +0000 UTC m=+68.115105139" watchObservedRunningTime="2026-10-31 04:04:59.935027568 +0000 UTC m=+68.115733863"
Oct 31 04:05:00.150650 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:00.150555    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:00.364027 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:00.363988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:02.906707 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:02.906665    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w49lj" event={"ID":"ac673d9c-d013-487f-828d-72c57b8386b7","Type":"ContainerStarted","Data":"4396a981474eff284ed57aaf211794dece4e92b61e9c248fab9ca6cfe387355e"}
Oct 31 04:05:02.907196 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:02.906792    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:05:02.949105 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:02.949050    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w49lj" podStartSLOduration=67.517395334 podStartE2EDuration="1m10.949031274s" totalImagesPullingTime="3.43163594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:02.948192084 +0000 UTC m=+71.128898380" watchObservedRunningTime="2026-10-31 04:05:02.949031274 +0000 UTC m=+71.129737569"
Oct 31 04:05:03.911626 ip-10-0-140-110 kubenswrapper[2571]: I1031 04:05:03.911535    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s4d8g" event={"ID":"4502cc34-9b28-4f7e-a13b-594221372bbf","Type":"ContainerStarted","Data":"3837f23871e20f6271a397e62508ac343198341661326c4c25d580e514f0e9f5"}
Oct 31 04:05:03.916971 ip-10-0-140-110 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 04:05:03.923962 ip-10-0-140-110 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 04:05:03.924126 ip-10-0-140-110 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 04:05:03.924326 ip-10-0-140-110 systemd[1]: kubelet.service: Consumed 4.894s CPU time.
Oct 31 04:05:03.935105 ip-10-0-140-110 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 04:05:04.031305 ip-10-0-140-110 kubenswrapper[6931]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:05:04.031305 ip-10-0-140-110 kubenswrapper[6931]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 04:05:04.031305 ip-10-0-140-110 kubenswrapper[6931]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:05:04.031305 ip-10-0-140-110 kubenswrapper[6931]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 04:05:04.033461 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033373    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.033461 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033460    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.033461 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033464    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033467    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033471    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033473    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033476    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033479    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033482    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033485    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033487    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033490    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033492    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033496    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033499    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033502    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033504    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033507    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033509    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033512    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033515    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033518    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.033558 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033520    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033525    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033529    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033532    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033535    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033543    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033547    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033549    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033552    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033555    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033558    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033561    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033564    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033567    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033570    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033573    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033576    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033578    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033581    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.034109 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033583    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033586    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033588    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033591    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033594    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033596    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033600    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033603    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033606    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033609    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033612    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033614    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033617    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033620    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033622    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033625    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033627    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033630    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033633    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033636    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.034620 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033639    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033642    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033644    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033647    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033650    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033652    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033655    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033658    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033660    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033663    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033666    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033669    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033671    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033674    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033676    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033679    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033682    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033685    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033688    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033691    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.035129 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033694    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033697    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033699    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033701    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033704    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033707    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033709    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033712    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033714    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033719    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033723    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033727    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033730    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033733    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033736    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033738    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033741    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033744    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033746    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.035674 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033749    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033751    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033754    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033756    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033759    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.033761    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034581    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034588    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034591    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034595    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034597    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034600    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034603    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034611    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034614    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034617    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034620    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034622    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034625    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034628    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.036166 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034631    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034634    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034637    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034639    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034643    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034647    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034650    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034653    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034656    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034658    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034661    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034664    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034667    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034669    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034672    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034675    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034677    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034680    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034682    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.036707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034685    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034688    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034690    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034693    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034695    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034698    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034700    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034704    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034707    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034710    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034712    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034715    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034717    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034720    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034723    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034726    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034729    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034732    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034734    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034737    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.037208 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034740    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034742    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034748    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034750    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034753    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034756    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034758    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034761    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034764    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034768    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034771    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034775    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034778    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034781    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034784    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034787    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034789    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034792    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034794    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.037707 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034797    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034800    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034803    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034806    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034808    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034811    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034813    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034816    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034819    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034822    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034824    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034827    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034829    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034832    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034835    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034842    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034845    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034847    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034850    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.038227 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034853    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034856    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034858    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034861    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034863    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034866    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034869    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034872    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034874    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034877    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034879    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034883    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034886    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034888    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.034891    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035420    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035427    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035431    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035434    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035437    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.038715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035440    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035443    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035446    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035449    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035451    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035454    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035457    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035459    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035463    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035465    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035468    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035470    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035473    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035476    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035479    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035481    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035484    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035488    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035491    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.039230 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035494    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035497    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035499    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035502    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035504    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035508    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035511    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035513    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035516    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035519    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035522    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035525    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035527    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035530    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035532    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035535    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035537    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035540    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035543    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.039752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035546    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035548    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035551    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035553    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035556    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035559    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035561    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035564    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035566    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035569    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035572    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035574    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035577    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035579    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035582    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035585    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035587    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035589    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035592    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.040235 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035595    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035597    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035600    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035603    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035608    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035611    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035614    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035616    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035619    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035621    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035624    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035626    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035629    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035631    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035633    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035636    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035638    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035641    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035644    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.040708 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035646    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035649    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035651    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035654    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035657    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035659    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035662    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035664    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035667    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035669    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035672    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035675    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035677    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035680    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035683    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035686    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035688    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035693    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035695    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035700    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.041424 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035703    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035706    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035708    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035711    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.035715    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035803    6931 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035815    6931 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035826    6931 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035832    6931 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035839    6931 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035844    6931 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035849    6931 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035853    6931 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035857    6931 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035860    6931 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035864    6931 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035867    6931 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035870    6931 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035873    6931 flags.go:64] FLAG: --cgroup-root=""
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035876    6931 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035879    6931 flags.go:64] FLAG: --client-ca-file=""
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035883    6931 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035886    6931 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035890    6931 flags.go:64] FLAG: --cluster-domain=""
Oct 31 04:05:04.042251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035893    6931 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035897    6931 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035900    6931 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035903    6931 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035907    6931 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035912    6931 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035916    6931 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035920    6931 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035923    6931 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035926    6931 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035929    6931 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035932    6931 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035935    6931 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035939    6931 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035942    6931 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035947    6931 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035950    6931 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035953    6931 flags.go:64] FLAG: --enable-server="true"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035956    6931 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035960    6931 flags.go:64] FLAG: --event-burst="100"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035963    6931 flags.go:64] FLAG: --event-qps="50"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035966    6931 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035970    6931 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035973    6931 flags.go:64] FLAG: --eviction-hard=""
Oct 31 04:05:04.042902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035977    6931 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035980    6931 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035983    6931 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035987    6931 flags.go:64] FLAG: --eviction-soft=""
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035990    6931 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035993    6931 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035996    6931 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.035999    6931 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036002    6931 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036005    6931 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036008    6931 flags.go:64] FLAG: --feature-gates=""
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036012    6931 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036015    6931 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036021    6931 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036024    6931 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036028    6931 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036031    6931 flags.go:64] FLAG: --help="false"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036034    6931 flags.go:64] FLAG: --hostname-override="ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036037    6931 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036040    6931 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036043    6931 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036046    6931 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036050    6931 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036054    6931 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 04:05:04.043533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036057    6931 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036060    6931 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036063    6931 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036066    6931 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036069    6931 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036072    6931 flags.go:64] FLAG: --kube-reserved=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036075    6931 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036078    6931 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036081    6931 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036084    6931 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036087    6931 flags.go:64] FLAG: --lock-file=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036090    6931 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036093    6931 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036096    6931 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036101    6931 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036104    6931 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036107    6931 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036110    6931 flags.go:64] FLAG: --logging-format="text"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036113    6931 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036117    6931 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036119    6931 flags.go:64] FLAG: --manifest-url=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036124    6931 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036128    6931 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036131    6931 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036136    6931 flags.go:64] FLAG: --max-pods="110"
Oct 31 04:05:04.044138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036139    6931 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036142    6931 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036145    6931 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036148    6931 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036151    6931 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036166    6931 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036171    6931 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036178    6931 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036182    6931 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036185    6931 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036189    6931 flags.go:64] FLAG: --pod-cidr=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036192    6931 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036195    6931 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036198    6931 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036201    6931 flags.go:64] FLAG: --port="10250"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036204    6931 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036207    6931 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008953caa12f115f7"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036210    6931 flags.go:64] FLAG: --qos-reserved=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036213    6931 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036216    6931 flags.go:64] FLAG: --register-node="true"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036219    6931 flags.go:64] FLAG: --register-with-taints=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036223    6931 flags.go:64] FLAG: --registry-burst="10"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036226    6931 flags.go:64] FLAG: --registry-qps="5"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036229    6931 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036232    6931 flags.go:64] FLAG: --reserved-memory=""
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036235    6931 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 04:05:04.044766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036238    6931 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036241    6931 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036246    6931 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036249    6931 flags.go:64] FLAG: --runonce="false"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036252    6931 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036255    6931 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036259    6931 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036262    6931 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036265    6931 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036268    6931 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036271    6931 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036274    6931 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036278    6931 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036281    6931 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036284    6931 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036287    6931 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036290    6931 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036293    6931 flags.go:64] FLAG: --system-cgroups=""
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036296    6931 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036301    6931 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036304    6931 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036307    6931 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036312    6931 flags.go:64] FLAG: --tls-min-version=""
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036315    6931 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036318    6931 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 04:05:04.045404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036320    6931 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036323    6931 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036327    6931 flags.go:64] FLAG: --v="2"
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036331    6931 flags.go:64] FLAG: --version="false"
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036335    6931 flags.go:64] FLAG: --vmodule=""
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036339    6931 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036342    6931 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036496    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036501    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036506    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036509    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036512    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036515    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036519    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036522    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036526    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036528    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036531    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036534    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036538    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.046041 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036541    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036543    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036546    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036549    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036551    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036554    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036556    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036559    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036562    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036565    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036568    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036571    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036573    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036576    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036579    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036581    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036584    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036587    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036589    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036592    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.046613 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036606    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036610    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036613    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036616    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036619    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036622    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036624    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036628    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036631    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036634    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036636    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036640    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036643    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036646    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036648    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036651    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036654    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036656    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036659    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.047366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036661    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036664    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036667    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036669    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036672    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036675    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036678    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036681    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036683    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036686    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036688    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036691    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036693    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036697    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036701    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036704    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036707    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036710    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036713    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036716    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.048015 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036719    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036722    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036724    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036727    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036731    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036734    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036736    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036739    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036742    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036744    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036747    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036749    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036752    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036755    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036758    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036762    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036765    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036768    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036770    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036773    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.048697 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036775    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036778    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036780    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036783    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036785    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036788    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036790    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036793    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036796    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036798    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036801    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036803    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036806    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036809    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.036834    6931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036869    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.049616 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036873    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036878    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036881    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036884    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036886    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036889    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036891    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036894    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036897    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036899    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036902    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036906    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036909    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036911    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036914    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036917    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036919    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036922    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036925    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036928    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.050369 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036931    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036933    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036936    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036938    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036941    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036943    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036946    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036949    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036952    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036954    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036957    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036960    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036963    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036967    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036970    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036972    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036975    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036978    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036980    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.051011 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036983    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036986    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036989    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036991    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036995    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.036998    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037000    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037003    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037005    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037008    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037011    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037013    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037016    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037019    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037021    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037024    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037028    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037031    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037034    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037037    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.051717 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037039    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037042    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037045    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037048    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037050    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037056    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037061    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037064    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037066    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037069    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037072    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037075    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037077    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037080    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037082    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037085    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037089    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037092    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037095    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037098    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.052322 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037100    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037103    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037105    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037108    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037110    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037113    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037115    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037118    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037121    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037123    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037126    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037128    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037131    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037133    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037136    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037138    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037141    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037144    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037151    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.052812 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037167    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037171    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037174    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037189    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037193    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037196    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.037199    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.042054    6931 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.042136    6931 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.042218    6931 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.042239    6931 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042549    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042564    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042569    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042572    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042575    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042579    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042582    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042585    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.053293 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042588    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042591    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042593    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042596    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042599    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042602    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042604    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042607    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042610    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042613    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042615    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042620    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042623    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042625    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042628    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042631    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042634    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042636    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042639    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042641    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.053783 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042644    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042649    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042653    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042656    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042659    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042662    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042665    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042667    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042670    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042673    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042676    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042678    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042683    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042686    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042689    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042692    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042694    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042697    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042700    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.054366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042703    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042706    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042708    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042711    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042714    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042716    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042719    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042722    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042725    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042728    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042730    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042733    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042736    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042739    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042742    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042745    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042747    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042750    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042753    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.054896 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042755    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042758    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042761    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042764    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042766    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042769    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042772    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042774    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042777    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042779    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042782    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042785    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042787    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042791    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042793    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042796    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042799    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042801    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042804    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042806    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.055484 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042809    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042811    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042814    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042817    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042819    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042822    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042825    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042827    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042830    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042832    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042835    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042838    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042840    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042843    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042845    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042848    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042851    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042853    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042856    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.056013 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.042859    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.042888    6931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043028    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043033    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043036    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043039    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043042    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043046    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043049    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043052    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043055    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043058    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043060    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043063    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043065    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043068    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.056502 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043070    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043073    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043075    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043078    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043081    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043083    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043086    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043088    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043091    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043094    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043096    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043099    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043102    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043104    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043107    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043109    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043112    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043114    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043117    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043119    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.056897 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043122    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043124    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043127    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043130    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043133    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043135    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043138    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043140    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043143    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043146    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043149    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043152    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043172    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043175    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043178    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043180    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043182    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043185    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043188    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.057479 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043190    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043193    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043196    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043198    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043201    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043204    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043207    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043210    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043212    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043215    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043217    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043220    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043223    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043226    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043228    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043231    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043233    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043236    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043239    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.057951 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043242    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043244    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043247    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043249    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043252    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043254    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043257    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043261    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043264    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043267    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043269    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043272    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043274    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043277    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043280    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043282    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043285    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043288    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043290    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.058442 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043293    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043295    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043298    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043300    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043303    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043305    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043308    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043310    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043313    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043315    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043318    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043322    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043326    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043329    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043332    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.043355    6931 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 04:05:04.058946 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043387    6931 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043391    6931 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043394    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043397    6931 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043400    6931 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043403    6931 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043405    6931 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043408    6931 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043411    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043416    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043419    6931 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043422    6931 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043424    6931 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043427    6931 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043430    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043433    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043435    6931 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043438    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043441    6931 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 04:05:04.059371 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043444    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043447    6931 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043449    6931 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043452    6931 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043455    6931 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043457    6931 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043460    6931 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043462    6931 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043465    6931 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043468    6931 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043470    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043474    6931 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043477    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043480    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043482    6931 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043485    6931 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043487    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043490    6931 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043493    6931 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043495    6931 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 04:05:04.059837 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043498    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043500    6931 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043504    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043507    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043509    6931 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043512    6931 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043514    6931 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043517    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043519    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043522    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043526    6931 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043530    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043532    6931 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043535    6931 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043538    6931 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043542    6931 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043547    6931 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043550    6931 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043553    6931 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 04:05:04.060392 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043557    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043560    6931 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043563    6931 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043566    6931 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043569    6931 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043573    6931 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043576    6931 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043578    6931 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043581    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043584    6931 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043586    6931 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043589    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043591    6931 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043594    6931 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043597    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043600    6931 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043603    6931 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043606    6931 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043608    6931 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043612    6931 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 04:05:04.061130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043616    6931 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043619    6931 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043622    6931 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043625    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043628    6931 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043631    6931 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043634    6931 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043636    6931 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043639    6931 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043641    6931 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043644    6931 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043647    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043649    6931 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043652    6931 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043654    6931 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043657    6931 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043659    6931 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043662    6931 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043665    6931 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043668    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 04:05:04.061902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043670    6931 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043673    6931 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043676    6931 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043678    6931 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043681    6931 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043683    6931 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043687    6931 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:04.043690    6931 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.043755    6931 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.044737    6931 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.044786    6931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.044963    6931 server.go:1014] "Starting client certificate rotation"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.045104    6931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.045148    6931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:58:52 +0000 UTC" deadline="2028-08-15 04:12:10.505895264 +0000 UTC"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.045188    6931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15696h7m6.460709208s"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.045413    6931 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 04:05:04.062576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.046510    6931 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 04:05:04.063026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.048190    6931 log.go:25] "Validated CRI v1 runtime API"
Oct 31 04:05:04.063026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.050713    6931 log.go:25] "Validated CRI v1 image API"
Oct 31 04:05:04.063026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.051468    6931 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 04:05:04.063026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.053917    6931 fs.go:135] Filesystem UUIDs: map[06da1b3c-7e10-4a4a-bd2c-7cdcbc520783:/dev/nvme0n1p4 4036d856-6e78-4628-9882-d04d37d31c5a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 04:05:04.063200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.053937    6931 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9/userdata/shm major:0 minor:516 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d/userdata/shm major:0 minor:522 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a979666eb6613c6097b776e7288cf2f9b4603eb00bbde9291dddcf00596c814/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a979666eb6613c6097b776e7288cf2f9b4603eb00bbde9291dddcf00596c814/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5/userdata/shm major:0 minor:513 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b/userdata/shm major:0 minor:425 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~projected/kube-api-access-d2vrn:{mountpoint:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~projected/kube-api-access-d2vrn major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57/volumes/kubernetes.io~projected/kube-api-access-j8k98:{mountpoint:/var/lib/kubelet/pods/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57/volumes/kubernetes.io~projected/kube-api-access-j8k98 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~projected/kube-api-access-vht87:{mountpoint:/var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~projected/kube-api-access-vht87 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/config-reloader/2 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/prometheus/15 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~empty-dir/config-out major:0 minor:417 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/kube-api-access-fhfhh:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/kube-api-access-fhfhh major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/tls-assets major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/config major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/web-config major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/alertmanager/9 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/kube-api-access-nzkxm:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/kube-api-access-nzkxm major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/tls-assets major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/config-volume major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/web-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~projected/kube-api-access-gx49j:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~projected/kube-api-access-gx49j major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4502cc34-9b28-4f7e-a13b-594221372bbf/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4502cc34-9b28-4f7e-a13b-594221372bbf/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:512 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~projected/kube-api-access-fsm8k:{mountpoint:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~projected/kube-api-access-fsm8k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~projected/kube-api-access-k6wsd:{mountpoint:/var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~projected/kube-api-access-k6wsd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~secret/metrics-certs major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95ce4d22-fc05-4428-a9dd-651a3b1c4686/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/95ce4d22-fc05-4428-a9dd-651a3b1c4686/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~projected/kube-api-access-z6wdt:{mountpoint:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~projected/kube-api-access-z6wdt major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac673d9c-d013-487f-828d-72c57b8386b7/volumes/kubernetes.io~projected/kube-api-access-c5lg9:{mountpoint:/var/lib/kubelet/pods/ac673d9c-d013-487f-828d-72c57b8386b7/volumes/kubernetes.io~projected/kube-api-access-c5lg9 major:0 minor:515 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf269a0-e3d7-4932-ba07-f4f48ee0c787/volumes/kubernetes.io~projected/kube-api-access-mn5w5:{mountpoint:/var/lib/kubelet/pods/acf269a0-e3d7-4932-ba07-f4f48ee0c787/volumes/kubernetes.io~projected/kube-api-access-mn5w5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~projected/kube-api-access-vrs52:{mountpoint:/var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~projected/kube-api-access-vrs52 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6e10b9e-7427-4d6d-9a99-e17824c49078/volumes/kubernetes.io~projected/kube-api-access-8sf9t:{mountpoint:/var/lib/kubelet/pods/b6e10b9e-7427-4d6d-9a99-e17824c49078/volumes/kubernetes.io~projected/kube-api-access-8sf9t major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~projected/kube-api-access-89nc9:{mountpoint:/var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~projected/kube-api-access-89nc9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~projected/kube-api-access-992wh:{mountpoint:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~projected/kube-api-access-992wh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca7768dc-fc1a-405d-92cc-75e17eaccf19/volumes/kubernetes.io~projected/kube-api-access-c9kmp:{mountpoint:/var/lib/kubelet/pods/ca7768dc-fc1a-405d-92cc-75e17eaccf19/volumes/kubernetes.io~projected/kube-api-access-c9kmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce70e1ef-553c-437b-bb60-f046bf05fa2e/volumes/kubernetes.io~projected/kube-api-access-lt257:{mountpoint:/var/lib/kubelet/pods/ce70e1ef-553c-437b-bb60-f046bf05fa2e/volumes/kubernetes.io~projected/kube-api-access-lt257 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~projected/kube-api-access-vbshb:{mountpoint:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~projected/kube-api-access-vbshb major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e57f88b0-c977-4b88-82b3-14f7b7792845/volumes/kubernetes.io~projected/kube-api-access-p4z92:{mountpoint:/var/lib/kubelet/pods/e57f88b0-c977-4b88-82b3-14f7b7792845/volumes/kubernetes.io~projected/kube-api-access-p4z92 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~projected/kube-api-access-qlmff:{mountpoint:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~projected/kube-api-access-qlmff major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~projected/kube-api-access-sbjvx:{mountpoint:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~projected/kube-api-access-sbjvx major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~projected/kube-api-access-gwbvn:{mountpoint:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~projected/kube-api-access-gwbvn major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~projected/kube-api-access-qkt4q:{mountpoint:/var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~projected/kube-api-access-qkt4q major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c9959ae695b7ccde39125a7d0974ebae0ff56fa83efefe15867d080dd1b32396/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b1272cc06e9332c56fe905d2829d504964bb8626f0ea273e5e63f9cf956814b1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a145a5ff35f788b3abff28e6b4d3505547f8606f84289dbb036d0f4d1d814dd0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/29137a57943f8ea6da76b7c7527193ca5411425f83dde9fdf91f7c7414679d5d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6f2f6c3037a115a3f855d31d8a9c45e0076a28a5409b35d44bd7d20888507f58/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0168ec530704a75989a2f7433a960bf791aec3743555c8f1259b604f143263f0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/27dac936acf4cd3d760f571d26e6b9fa1d58553cf42c981733be30e3dc2a2efc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/46e562e49a7b03f3b66277bb8a685ebc6c54c5c19f5f4e7306d6bb555f9f2883/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7b6be46e7c2512da1ab710c516ff6c342c2736e15aee23678752ebcf405d22cf/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/4b9e5eb4c983e7eb8ce4780ab01b08f4bb09307976535a97e68590eba9d117c6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8f919d765d027801b4f78185ff04d9fe51fc484da7de3092fe31bfc0518f3631/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/c75f9d9320e7313f71238fbf8e90ad4377235fed51c6310a9a5715764b3c9d80/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b4e9c3136d478ae59b3cabe9e5b68e3d77fe5e9f87dc40e08e0a6cf5603704ed/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/17801667a25d8956088c262786491fa5ecad98fcb431e38938edff8007b2e434/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8da04d281faab09101a0a751116bf5eea0871960b4448494674575e662720cce/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f97f4f0cab76d27664e00e1d67308df6b36ae18646f556e517d363710f834e45/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/19a0302223addd75ad4aaddf798bb124f0567bc257181882f5fd2c3b4ee0123a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/72d0718db83bdf1f0bd8383a627888674f828233563e3d50e61c297f10fe9eed/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9c1a6539c30935208f6afbff7a5a198cf7b3f9ccd3ed40240bb681fc318dfffe/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/06bd70fe198c7ad5015bf019a603749a2ee247577b01b8f662c0fbfd9febc66b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/998cac66fb174f87c77d2f22dcdbf48c2a49351e16b2a08a8e2cb7005a1b3192/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/134e8922f286453523bbd9742362ff21f73989f7124ac4b2ebf323f9e7894907/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/d06d468305c43abd45f5c7f17371de6340f7f880714da2be48e352a8139d9a9d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/cb305d2cec30f6e113c28df7933abd656d2d5a231c3115347437da6c8350153b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5b544ecc03daebd9dfc21bb0ceea1dc7ff10d5e530c08937fd1d23587001baed/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2e8615296e90950f929d94b173183b08ba3bbed0ba516dbb3defd451cc2aaac7/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/830e6a98a2e5fbf399f78cbacba2d7c6aaa50757191bd55fac77e4b090617812/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b6ec0d684460cc1173eed7a9b0f172af85b9add457fa732fda53d1f4507b1bc3/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c960a1ef027c5ec62fde872f828a0760e26bce238c9216999d7bfe3b8a6d3bd2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1dba02993bafa48005b290e57b863b3b2eed94c9d296452da4b0c26718c0de00/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8ed851781e2706c3953362b9cd777eea674c9e58fc61af630cd8b8a6a9c480e9/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e83c6b1648b0376b6d184704773b79d92b56dcad8a7f1ffddd42caf6c3d6ab59/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/ae2e2c8591f33c44e69e551a8b9443b60e749b04c0b5d38022a827b142021a0b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b0e95611c4049c35c5aa2d31a317a483f4c27022c58edc123a7cb29e572e5946/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/aa48381c641bb88dd34630348ff48851fd0e5b664b416d876a3b05b7964a1571/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/108178d89cf28766dcbc53ab0b1b9d3fbad8081e30315e66ab6f318b394f3f78/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/2a2101b80622adf30ce7c49df165c65e96b5168981a4bf95b9e638b05b53475e/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/34f44caeb0cb7ee84c061e0c00c2376d5637c5a8bdb35b3d07179b92fea916cf/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/f5dc17f05d97e77c507334871b1da5744d28c01fab7131586d04f0200219ed53/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/a678581ad61e6d73ec6b83fa6b00c07bfe9df0c3b69fd2927f9d3d37850c2c1e/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/427b6f47832f070f28ef7d5338633ba7cf339fb760c20799240aeefc47269030/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/f49dd2ea85b47e4d9dbc4a1bd4daf8bd73f28d0f7c6c66a2fe4e9635e2600b29/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/5fc184f41ceeea6a546f403a7d1a0a1e5e7ac125aad9a712eefde615733be11c/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/3784d0e0f684669f1648c5393c552371b4c8f9f44e6608fc632bc0dba1e15783/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4ce1973fd99ad28c132915d51d084ee0f50dca5928bcf8fffd5bfd358e5b0d04/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/3a0f260a860a41afa7614ad332197e0d35d97b20feb1ed8ab95231c7822f4401/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/34652bfaaf3d7ff964f13640f9a658567a20d914df61aca2d511ce5e88bdfb6b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/433974198ec79ae78228595311c27205166c3f6de2569d2807e607f4e897a457/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/c5f5e163dce2bc59534305ffa1598a72b7b3b3a0dbdd23a7e647c4797de5f9ac/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/0e337fa9bdba0740ebd28a9f314546bb0f828ae12e9683ee8d72ecf0d9bf5138/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/cab140de0dcd6e027800df3239e60471af4fa4108af60752231dfccf8f6342c9/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/fca423aa808cbe262cc05ac5104bdbb2a79bbebc5bf9ef811c4f4fdfb687d8cc/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/ece2a9ef591f109505dafc1d0019251a09983ac168d6c7fff4426f8e706b514b/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/a2221ce9afb0e449facc9634a6ff8a3288abbcbb6f0e4a1ce1f0aabe00e388b0/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/69f7658b6e1925b94adb6f46731fe3c440087843c4b0808c08b3856dc02c42b5/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/686a53b0592445bf627a21b7a18b2014b7b5fe9c38d1008e006b4982826403a7/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/fc0194960b77bd072ad86f34e214c9aa2700bf4a35b4b65eb976af4d855785b7/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/8b0866d8c9b1ff1d38ef816b3c5dffb3d8655dd7c47fa159ae66f165b4846130/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/06f35233320b40f6547a7f0e22abcf0db2020bbd4057159127b1886067e350a1/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/768d5bdb323edf1c62e21ece23c41174d930bdfe0cab5b6e8e3d93dbedab9725/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/003fa783eada6f53bd9627085e461d489f34e924388330f667d3b82f5666d8f5/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/4edd592b95ed42f79187db7194b24c48b3ced5eda96fe000278ab7240378a544/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/60218d65a2b21cc5acb72c41e0bca3f82a9468448a1dd1179e52692188d4900d/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/9709687a1d88f501fff23d4621965f279eca395f7d72e6b666cd9bc728bdb5e8/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/06cf7732eb0d6db33d71c90e5514eb779c82a17dcf6fee55d18667d38f9c5c5f/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/ddf367c4a82df85dd3970566a22d21fc2d4133752db0898a5cabc6f033ae1550/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/759877ddc51f0ceb7189e77bd88f449e66b0d14c82ea4f112573060f7163d395/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/821b72599775bd9412672034fbd1a425c9feaf099b5c9b3bcb539e681cdfb785/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/3a3420d1c41bb34cebe52626187e9639f4bbc7fdb7783321d9beedf88041cb9f/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/0618ded4997363bccd445b372d54016d0156926a4a8eeb645c066e50b5bcaf4f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/3a7211aa57b8f1d2c6066c3bc6ccb5271a920b70754263bf7a3d37a48e00b51b/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/978de7c6da6df049bc8083700da719be2002b9c766961b8dc1aa80280088c15c/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/4401a79b65b3b4692459259a35add5d3192294c17a3ad5c7e50e3af3c58e1b90/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/c87ed81cac6139326be8bee096c438520eb4c7999f517459eaf9075e929921ac/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/d1ff29e42414026856b6a7b181c4f1c036635907b651a149ed43b2a7f15e5e91/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/d39ef5546d0fd6a4183d28642a48775154fcd396062f92d7c73d90a6c4596d72/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/23bf77e008f94093a012afa1c8662b846ecac399e4857970240c1b6e1af7c3a7/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/1e8c9137fe333fa1461c066c10cfcd39baeeb6b077902a368358dc5b06576494/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5d864a045b609e6590c9c9e002ef7dfaf290900e7d863585fa5572b3860aaba6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/051675dd3e7a815ce048c816623e473698e363e34c6516127939944598ce4810/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/a4aacac1fe603bafb9e3c401b72bed4c827f1c5ff2c7f7bf5dab24a49f6ffcff/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/7e5f4f8c10bfba30240b7850018dd82743d0fa07d6dcbe9b900a6e20b3691b25/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/856239699d7c2c79c48a933367bc19ea81980d7fc50ba0f4ac1048a39b3266a5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e9e1bcc5d69b5de39f14fb08ae0a3dde91e88b3275bc75d03ff7517f38034b07/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b8f6b2461648ce799ef9bb8b5f755b72e16ad10a52d12cc4f5e6ff414329c19c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1251f04b61605ef36d123be2934988183f2572147ff865b661506eda7aa8b23c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e5141eb3a607dabf3c92225b885b0e16a142620ef3c912978641a9be11e4cb84/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/eab3fbc464f569fce070fd83c24fd6fdcac675140a42e87319ed1d0eabb7f7fa/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a696636c94c8397b32f11b74f20a1d9f311635fa60a2ec7f557bcec8ad0127fb/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 04:05:04.064326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.063846    6931 manager.go:217] Machine: {Timestamp:2026-10-31 04:05:04.063391005 +0000 UTC m=+0.096390675 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108883 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7202890d9c05b4e7c37cd61900e1 SystemUUID:ec2f7202-890d-9c05-b4e7-c37cd61900e1 BootID:ea258aa9-96ca-4867-a474-06e311dd9703 Filesystems:[{Device:/run/containers/storage/overlay-containers/3fe67f4fcda24c7b0619ce68e27787d2fdd51cfe8c11537aa2f8b5b62bf0d1d0/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f0dc6e0a5774f09dc12c58ad25a7c84941584a312d6b0cf6764751610e8f764/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/588402c273330e3dfa3a06739281e5e95df108dd0bffd647e9a12d3a78e075dc/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a979666eb6613c6097b776e7288cf2f9b4603eb00bbde9291dddcf00596c814/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de3dd0dc1d3237500519e4abf91656c7ca804ea77853732cf930190eb98f54f8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~projected/kube-api-access-vbshb DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95ce4d22-fc05-4428-a9dd-651a3b1c4686/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~projected/kube-api-access-z6wdt DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfc78a21c7b20179101c68b3bcc656aa00b849f36926503b9bafcdc359de3897/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/kube-api-access-fhfhh DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d10f599b0e14734be65b9167d40a8d37ed6d692a529f2e1d01787ea726e0d44f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:419 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~projected/kube-api-access-d2vrn DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~projected/kube-api-access-992wh DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:413 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0b13fdc963e10bdf147c6563a5ff3823152d4802884fdeee48cff7f13175420/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~projected/kube-api-access-89nc9 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6e10b9e-7427-4d6d-9a99-e17824c49078/volumes/kubernetes.io~projected/kube-api-access-8sf9t DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~projected/kube-api-access-gx49j DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32633e94e7f23a988f5ba3af371927981de6fb1404077d3637fa2da112af49d9/userdata/shm DeviceMajor:0 DeviceMinor:516 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:412 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c669224f438cda3c7642f5c5b730fe263b54cd49f1b0e19e40364f1c2a5965c5/userdata/shm DeviceMajor:0 DeviceMinor:513 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acf269a0-e3d7-4932-ba07-f4f48ee0c787/volumes/kubernetes.io~projected/kube-api-access-mn5w5 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee586c905d7a5a2b82510b69b9d1a80e2a992917b0e4599bd737f38ac0889129/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e57f88b0-c977-4b88-82b3-14f7b7792845/volumes/kubernetes.io~projected/kube-api-access-p4z92 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4502cc34-9b28-4f7e-a13b-594221372bbf/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:512 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb6c8de0-72bd-40d8-aa86-34805cf518d0/volumes/kubernetes.io~projected/kube-api-access-qlmff DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac673d9c-d013-487f-828d-72c57b8386b7/volumes/kubernetes.io~projected/kube-api-access-c5lg9 DeviceMajor:0 DeviceMinor:515 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5d7df10-6133-426c-a8e1-48c78cdc1f44/volumes/kubernetes.io~projected/kube-api-access-vrs52 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43dcec0b26cfaca0eec4b009e24eba817d7505946afc6ea5df52dd8a14868095/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~projected/kube-api-access-gwbvn DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9d6393a80a7ddd67f0ac2da9f0222a068846259c3fa3884a5a8cddf5956b026/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:411 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:508 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ca7768dc-fc1a-405d-92cc-75e17eaccf19/volumes/kubernetes.io~projected/kube-api-access-c9kmp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f015bdf183270a12b2d49ab55c76a79f3201ed98b75e714771d85632ce388db/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f79174e7-f9ee-4451-a10f-c8ffb4b4f948/volumes/kubernetes.io~projected/kube-api-access-qkt4q DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c83f2509-20b7-4ece-9477-abeda7a82df1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes/kubernetes.io~projected/kube-api-access-nzkxm DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f47842abd8de54b8a090cf72eb82e576aa138461a919492310f2f434884dc91/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:416 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c15711-14f6-494e-a518-478f433cfeb1/volumes/kubernetes.io~projected/kube-api-access-k6wsd DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de8cf693-3022-4ad6-a50a-82466ad009ba/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68bf59fde2ce50095bfa8f4ee697c61717cff03a4709dbd9cff3e4a100394477/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436de514057e7faabbc2e0378c4eb81f3c3283e2df72ffb6c3cb1a047aaade0d/userdata/shm DeviceMajor:0 DeviceMinor:522 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57/volumes/kubernetes.io~projected/kube-api-access-j8k98 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~projected/kube-api-access-fsm8k DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~projected/kube-api-access-sbjvx DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:417 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f60643a3-3f4e-4308-a52e-0893b0fce7eb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48196ffe5fdf69add3c340b3988983cc5b13d34a1032ce230e3705deeced17db/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/030672d106c80f1a5587cf0ef248807600c25585e2c956c0e3771005c666b00a/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18aee04cb7250780ebf6c4eba7d4f91cab3bf746db4423fa825c2924cca7ced7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a345e1d5-f60a-473a-9cef-11db043e79df/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1fbb956a-7849-497a-9463-f85599ef4d57/volumes/kubernetes.io~projected/kube-api-access-vht87 DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35f518cc-282f-4dc6-9279-4e162eba47ab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:414 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:415 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b/userdata/shm DeviceMajor:0 DeviceMinor:425 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca197fb2-de4b-4936-9af4-e1177179830c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:407 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5460fbe1-498a-4aae-b32d-b8d97666d7b0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce70e1ef-553c-437b-bb60-f046bf05fa2e/volumes/kubernetes.io~projected/kube-api-access-lt257 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dad57d1523f115e22795f6696823fb88cf6a804771a32af9e190cf6234991ec3/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1cfd2f007cc9338 MacAddress:c2:72:16:f4:71:97 Speed:10000 Mtu:8901} {Name:1f015bdf183270a MacAddress:ca:17:19:54:a3:35 Speed:10000 Mtu:8901} {Name:32633e94e7f23a9 MacAddress:b6:e4:11:1e:7a:04 Speed:10000 Mtu:8901} {Name:3fe67f4fcda24c7 MacAddress:52:2b:81:18:f3:ee Speed:10000 Mtu:8901} {Name:436de514057e7fa MacAddress:92:f4:ce:f7:87:62 Speed:10000 Mtu:8901} {Name:43dcec0b26cfaca MacAddress:46:3e:e4:5e:c3:2b Speed:10000 Mtu:8901} {Name:48196ffe5fdf69a MacAddress:8a:9b:f1:82:8c:a4 Speed:10000 Mtu:8901} {Name:588402c273330e3 MacAddress:12:d9:dc:af:fb:10 Speed:10000 Mtu:8901} {Name:9f0dc6e0a5774f0 MacAddress:fa:30:fc:48:1e:2a Speed:10000 Mtu:8901} {Name:9f47842abd8de54 MacAddress:56:d9:06:06:c9:63 Speed:10000 Mtu:8901} {Name:ad700e398906461 MacAddress:5a:c8:0d:0e:06:ec Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:df:06:e3:cf Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:3b:02:b4:1c:f1 Speed:0 Mtu:8901} {Name:c669224f438cda3 MacAddress:22:b6:3e:a0:dc:3b Speed:10000 Mtu:8901} {Name:cb8c3529edab7c5 MacAddress:a2:00:54:3c:c6:30 Speed:10000 Mtu:8901} {Name:d9d6393a80a7ddd MacAddress:02:dc:36:f6:0d:87 Speed:10000 Mtu:8901} {Name:dad57d1523f115e MacAddress:02:47:ff:82:18:66 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:df:06:e3:cf Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:88:e4:5b:20:df Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:be:4a:4c:c0:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 04:05:04.064702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064330    6931 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 04:05:04.064702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064378    6931 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 04:05:04.064818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064756    6931 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 04:05:04.064959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064793    6931 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 04:05:04.065039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064962    6931 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 04:05:04.065039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064975    6931 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 04:05:04.065039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.064990    6931 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 04:05:04.065039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065017    6931 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 04:05:04.065229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065131    6931 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 04:05:04.065359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065345    6931 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 04:05:04.065479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065468    6931 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 04:05:04.065538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065516    6931 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 04:05:04.065538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065536    6931 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 04:05:04.065631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065550    6931 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 04:05:04.065631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065564    6931 kubelet.go:405] "Adding apiserver pod source"
Oct 31 04:05:04.065631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.065576    6931 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 04:05:04.066357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.066341    6931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 04:05:04.066357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.066356    6931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 04:05:04.066873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.066854    6931 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 04:05:04.069742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.069724    6931 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 04:05:04.069742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.069739    6931 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 04:05:04.069889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.069870    6931 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 04:05:04.070093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070077    6931 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 04:05:04.070171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070118    6931 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 04:05:04.070259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070248    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070267    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070279    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070288    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070297    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070307    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 04:05:04.070315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070316    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070325    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070336    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070345    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070358    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070371    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070416    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 04:05:04.070637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.070426    6931 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 04:05:04.073844 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.073828    6931 server.go:1263] "Started kubelet"
Oct 31 04:05:04.073933 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.073888    6931 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 04:05:04.074390 ip-10-0-140-110 systemd[1]: Started Kubernetes Kubelet.
Oct 31 04:05:04.074657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.074583    6931 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 04:05:04.074755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.074695    6931 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 04:05:04.075054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.075036    6931 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 04:05:04.075701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.075687    6931 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 04:05:04.083508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083486    6931 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 04:05:04.083625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083508    6931 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 04:05:04.083625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083558    6931 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:58:54 +0000 UTC" deadline="2028-05-22 00:59:55.131711718 +0000 UTC"
Oct 31 04:05:04.083625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083579    6931 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13652h54m51.048135404s"
Oct 31 04:05:04.083625 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.083622    6931 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Oct 31 04:05:04.083818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083627    6931 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 04:05:04.083818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083642    6931 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 04:05:04.083818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.083732    6931 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 04:05:04.085859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085815    6931 factory.go:153] Registering CRI-O factory
Oct 31 04:05:04.085859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085838    6931 factory.go:223] Registration of the crio container factory successfully
Oct 31 04:05:04.086023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085897    6931 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 04:05:04.086023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085907    6931 factory.go:55] Registering systemd factory
Oct 31 04:05:04.086023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085917    6931 factory.go:223] Registration of the systemd container factory successfully
Oct 31 04:05:04.086023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085936    6931 factory.go:103] Registering Raw factory
Oct 31 04:05:04.086023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.085956    6931 manager.go:1199] Started watching for new ooms in manager
Oct 31 04:05:04.086590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.086565    6931 manager.go:319] Starting recovery of all containers
Oct 31 04:05:04.087631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087592    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 31 04:05:04.087631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087634    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config" seLinuxMountContext=""
Oct 31 04:05:04.087631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087649    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a345e1d5-f60a-473a-9cef-11db043e79df" volumeName="kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087663    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83f2509-20b7-4ece-9477-abeda7a82df1" volumeName="kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087675    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087687    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4502cc34-9b28-4f7e-a13b-594221372bbf" volumeName="kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087699    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ce4d22-fc05-4428-a9dd-651a3b1c4686" volumeName="kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087711    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac673d9c-d013-487f-828d-72c57b8386b7" volumeName="kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087725    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087741    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087754    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087764    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087778    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087789    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a345e1d5-f60a-473a-9cef-11db043e79df" volumeName="kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087817    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087830    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087842    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 04:05:04.087882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087880    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087894    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087908    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e10b9e-7427-4d6d-9a99-e17824c49078" volumeName="kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087919    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087936    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087961    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087972    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087984    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.087996    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088010    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5460fbe1-498a-4aae-b32d-b8d97666d7b0" volumeName="kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088037    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d7df10-6133-426c-a8e1-48c78cdc1f44" volumeName="kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088055    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79174e7-f9ee-4451-a10f-c8ffb4b4f948" volumeName="kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088067    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a1ddf5-4fd2-4b73-bf05-9e15e10aba57" volumeName="kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088079    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a345e1d5-f60a-473a-9cef-11db043e79df" volumeName="kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088092    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088104    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088118    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088129    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088140    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d7df10-6133-426c-a8e1-48c78cdc1f44" volumeName="kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088151    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088176    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a1ddf5-4fd2-4b73-bf05-9e15e10aba57" volumeName="kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088190    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088203    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088214    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088225    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d7df10-6133-426c-a8e1-48c78cdc1f44" volumeName="kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088270    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57f88b0-c977-4b88-82b3-14f7b7792845" volumeName="kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088283    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088297    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088309    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e10b9e-7427-4d6d-9a99-e17824c49078" volumeName="kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088321    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088333    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088345    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088358    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088373    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088389    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088406    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79174e7-f9ee-4451-a10f-c8ffb4b4f948" volumeName="kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088419    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088433    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a1ddf5-4fd2-4b73-bf05-9e15e10aba57" volumeName="kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088446    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088471    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca7768dc-fc1a-405d-92cc-75e17eaccf19" volumeName="kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088486    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088501    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088513    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088529    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5460fbe1-498a-4aae-b32d-b8d97666d7b0" volumeName="kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088541    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca197fb2-de4b-4936-9af4-e1177179830c" volumeName="kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088552    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088567    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fbb956a-7849-497a-9463-f85599ef4d57" volumeName="kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088578    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088591    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088602    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088613    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ce4d22-fc05-4428-a9dd-651a3b1c4686" volumeName="kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088624    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce70e1ef-553c-437b-bb60-f046bf05fa2e" volumeName="kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088636    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.088616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088647    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fbb956a-7849-497a-9463-f85599ef4d57" volumeName="kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088660    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c15711-14f6-494e-a518-478f433cfeb1" volumeName="kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088672    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e10b9e-7427-4d6d-9a99-e17824c49078" volumeName="kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088687    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088702    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088716    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088728    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088740    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088754    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca197fb2-de4b-4936-9af4-e1177179830c" volumeName="kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088768    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088783    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a345e1d5-f60a-473a-9cef-11db043e79df" volumeName="kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088807    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088823    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088839    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088856    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5460fbe1-498a-4aae-b32d-b8d97666d7b0" volumeName="kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088868    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088880    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088926    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf269a0-e3d7-4932-ba07-f4f48ee0c787" volumeName="kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088945    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e10b9e-7427-4d6d-9a99-e17824c49078" volumeName="kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088957    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca197fb2-de4b-4936-9af4-e1177179830c" volumeName="kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.088974    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089003    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c15711-14f6-494e-a518-478f433cfeb1" volumeName="kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089017    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a" volumeName="kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089078    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089091    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089105    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089131    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089193    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089208    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089234    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089251    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089263    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f60643a3-3f4e-4308-a52e-0893b0fce7eb" volumeName="kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089288    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089318    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de8cf693-3022-4ad6-a50a-82466ad009ba" volumeName="kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089336    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089392    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" volumeName="kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089408    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5d7df10-6133-426c-a8e1-48c78cdc1f44" volumeName="kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089423    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83f2509-20b7-4ece-9477-abeda7a82df1" volumeName="kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089452    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089464    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089527    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca197fb2-de4b-4936-9af4-e1177179830c" volumeName="kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089556    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e57f88b0-c977-4b88-82b3-14f7b7792845" volumeName="kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089602    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb6c8de0-72bd-40d8-aa86-34805cf518d0" volumeName="kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089650    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79174e7-f9ee-4451-a10f-c8ffb4b4f948" volumeName="kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089679    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79174e7-f9ee-4451-a10f-c8ffb4b4f948" volumeName="kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089692    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb56708-d5b3-47bc-9bb6-1d931829bc34" volumeName="kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089703    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="337b4d1b-8c2b-4ba1-a463-38484614334c" volumeName="kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089719    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f518cc-282f-4dc6-9279-4e162eba47ab" volumeName="kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089726    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf269a0-e3d7-4932-ba07-f4f48ee0c787" volumeName="kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089741    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca197fb2-de4b-4936-9af4-e1177179830c" volumeName="kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089779    6931 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca7768dc-fc1a-405d-92cc-75e17eaccf19" volumeName="kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp" seLinuxMountContext=""
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089798    6931 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 04:05:04.091201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.089803    6931 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 04:05:04.112545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.112494    6931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 04:05:04.114270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.114241    6931 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 04:05:04.114270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.114269    6931 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 04:05:04.114518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.114295    6931 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 04:05:04.114518 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.114338    6931 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 04:05:04.126044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.126015    6931 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:05:04.126241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.126060    6931 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 04:05:04.126368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.126351    6931 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:05:04.134594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.134546    6931 generic.go:356] "Generic (PLEG): container finished" podID="f60643a3-3f4e-4308-a52e-0893b0fce7eb" containerID="5bf300c98e417c02fc5e652fda8f6cea54bba355898d988e5e42666c27999528" exitCode=0
Oct 31 04:05:04.149627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.149596    6931 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:05:04.149786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.149670    6931 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 04:05:04.160426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.160395    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.169987    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="e89fdd7f791029b713ad7754dfe2b6238c12f34881644f2b2fdfdcd1034b46f5" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.170027    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="ef262139e74a5bfa8af9b13e2d8b7edeece050ce1aa26a82858c64b997c53ce2" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.170037    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="cf33b53f1641bb69002728bca642b5d4d858b166cdc141f93683515fb6ba36f9" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.170045    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="d11c1218053f905ce03a48f7896e10a97cf595ea560de2a98d6a749893bbcf4a" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.170053    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="e6e3fa0fdc7106d91e9665146f2667ed30cfc9a689052bb76e369632bfc208a6" exitCode=0
Oct 31 04:05:04.170076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.170061    6931 generic.go:356] "Generic (PLEG): container finished" podID="b6e10b9e-7427-4d6d-9a99-e17824c49078" containerID="f55a3aed5f02f12405ffceefca58d38d9b56267be40fde8e51693ba1667e6fcd" exitCode=0
Oct 31 04:05:04.172540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.172511    6931 generic.go:356] "Generic (PLEG): container finished" podID="3cd74d1416e17adb1d88ce1b2d585255" containerID="715644b5e2af6b0c1d7bef4e1cf556a4d7911e41368fcbe56916d02f9d75a05c" exitCode=0
Oct 31 04:05:04.178316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.178293    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="eb0af5a40286d72459810c7bd19b4357a25fc11b71d96369fbabfd6433e26ed1" exitCode=0
Oct 31 04:05:04.214477 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.214448    6931 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 04:05:04.216372 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.216353    6931 manager.go:324] Recovery completed
Oct 31 04:05:04.242941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.242912    6931 cpu_manager.go:226] "Starting" policy="none"
Oct 31 04:05:04.242941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.242930    6931 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 04:05:04.242941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.242949    6931 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243071    6931 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243082    6931 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243099    6931 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243104    6931 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243109    6931 policy_none.go:50] "Start"
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243118    6931 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 04:05:04.243147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243127    6931 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 04:05:04.243376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243261    6931 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 04:05:04.243376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243269    6931 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 04:05:04.243376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.243274    6931 policy_none.go:44] "Start"
Oct 31 04:05:04.248136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.248107    6931 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 04:05:04.253276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.253259    6931 manager.go:341] "Starting Device Plugin manager"
Oct 31 04:05:04.253717 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.253290    6931 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 04:05:04.253717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.253718    6931 server.go:87] "Starting device plugin registration server"
Oct 31 04:05:04.253969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.253959    6931 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 04:05:04.254002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.253970    6931 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 04:05:04.254055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.254042    6931 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 04:05:04.254187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.254134    6931 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 04:05:04.254258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.254198    6931 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 04:05:04.254258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.254203    6931 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 04:05:04.254349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.254247    6931 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T04:05:04.254185058Z","UUID":"63c7c96e-49f9-45e1-b0f7-ca169d73390d","Handler":null,"Name":"","Endpoint":""}
Oct 31 04:05:04.255767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.255746    6931 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 04:05:04.255879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.255786    6931 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 04:05:04.255934 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.255892    6931 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 04:05:04.355085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.355037    6931 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 04:05:04.357118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.357101    6931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 04:05:04.357255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.357130    6931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 04:05:04.357255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.357139    6931 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Oct 31 04:05:04.357255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.357245    6931 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.393935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.393917    6931 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.394010 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.394000    6931 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.415496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.415445    6931 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"]
Oct 31 04:05:04.416094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416079    6931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.416147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416080    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"50080b7deadb8e30dee000b7f0bbadfa","Type":"ContainerStarted","Data":"75eda65653178c60d3f20f2189774dff8f7fdaf637f27dc72292ed0d1a220e8d"}
Oct 31 04:05:04.416147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416119    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"50080b7deadb8e30dee000b7f0bbadfa","Type":"ContainerStarted","Data":"45b61ef14a778dd69f8d7ecfb1b67cc2d349c3d89b1e10918523404832a5c63c"}
Oct 31 04:05:04.416254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416215    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerStarted","Data":"6de74c6023bb68d8b9b5aa5520eca8ed10a4930b6f09d2a851fd78ecd59f744e"}
Oct 31 04:05:04.416254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416233    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerDied","Data":"715644b5e2af6b0c1d7bef4e1cf556a4d7911e41368fcbe56916d02f9d75a05c"}
Oct 31 04:05:04.416254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416246    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"3cd74d1416e17adb1d88ce1b2d585255","Type":"ContainerStarted","Data":"dc098c4385a689344e4f0d07280cf48eb8086d3461f9d863cf0817ed71e46718"}
Oct 31 04:05:04.416350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.416320    6931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.441200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.441112    6931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:05:04.441200 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.441175    6931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.441687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.441672    6931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:05:04.441730 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:04.441711    6931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.528068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.528031    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.528233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.528068    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.528233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.528104    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628610    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628645    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628662    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628719    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628769    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50080b7deadb8e30dee000b7f0bbadfa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"50080b7deadb8e30dee000b7f0bbadfa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:04.628834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:04.628820    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3cd74d1416e17adb1d88ce1b2d585255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"3cd74d1416e17adb1d88ce1b2d585255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:05.066299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.066262    6931 apiserver.go:52] "Watching apiserver"
Oct 31 04:05:05.119935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.117612    6931 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 04:05:05.119935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.119673    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4","openshift-dns/dns-default-v6nhz","openshift-insights/insights-runtime-extractor-jphhc","openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-phzfq","openshift-network-operator/iptables-alerter-tqzjj","openshift-cluster-node-tuning-operator/tuned-rsdcv","openshift-ingress-canary/ingress-canary-gqp52","openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56","openshift-monitoring/telemeter-client-b8d6659cd-4n7qm","openshift-ovn-kubernetes/ovnkube-node-vg88m","kube-system/konnectivity-agent-gk4cs","openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d","openshift-network-diagnostics/network-check-target-w49lj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c","openshift-multus/multus-kgjcj","openshift-multus/network-metrics-daemon-dk458","openshift-monitoring/monitoring-plugin-77f4f76594-brlvf","kube-system/global-pull-secret-syncer-s4d8g","openshift-console/console-8594bc5d5f-sn56k","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/thanos-querier-68c888f8cf-rcbzw","openshift-image-registry/node-ca-xvz9v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-8rscm","openshift-monitoring/node-exporter-qfnsk","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"]
Oct 31 04:05:05.136660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136618    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lwzcw\""
Oct 31 04:05:05.136660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136660    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjn2p\""
Oct 31 04:05:05.136886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136663    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vc7dp\""
Oct 31 04:05:05.136886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136789    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xzg5s\""
Oct 31 04:05:05.136886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136663    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 04:05:05.137020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.136151    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 04:05:05.138063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138041    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 04:05:05.138224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138127    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 04:05:05.138287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138224    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-brpbt\""
Oct 31 04:05:05.138338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138292    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-brbhc\""
Oct 31 04:05:05.138419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138404    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9876q\""
Oct 31 04:05:05.138705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138689    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 04:05:05.138850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138837    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.138919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.138907    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 04:05:05.139028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139014    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.139114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139102    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.139243 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139230    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 04:05:05.139325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139313    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.139431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139418    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.139515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139503    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 04:05:05.139614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139601    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.139718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139705    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 04:05:05.141181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139811    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.141181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.139889    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.141181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.140013    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 04:05:05.143773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.143752    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 04:05:05.144024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144004    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 04:05:05.144134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144082    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45qpm\""
Oct 31 04:05:05.144134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144107    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfpck\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144134    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144099    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144178    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144240    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tnrrl\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144243    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 04:05:05.144271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144283    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 04:05:05.144493 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144318    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 04:05:05.144493 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144464    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 04:05:05.144592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144493    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldktz\""
Oct 31 04:05:05.144592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144522    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 04:05:05.144592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144527    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qzkqw\""
Oct 31 04:05:05.144592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144540    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 04:05:05.144592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144550    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144600    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfl64\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144618    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144637    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gl9k\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144689    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5nqr4\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144729    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w2pm\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144741    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctbtg\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144754    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144771    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144811    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144815    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.144828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144824    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gw6dk\""
Oct 31 04:05:05.145216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144851    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.145216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144884    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.145216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144939    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 04:05:05.145216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.144953    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 04:05:05.145637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145622    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 04:05:05.145697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145624    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.145752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145706    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.145752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145706    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-87lk2\""
Oct 31 04:05:05.145752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145731    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.145752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145752    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 04:05:05.145913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145869    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 04:05:05.145962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145918    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 04:05:05.145962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145922    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 04:05:05.145962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145947    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.145962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145956    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145961    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145967    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145928    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.145965    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.146028    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.146093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.146031    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 04:05:05.146337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.146317    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 04:05:05.147678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.147662    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 04:05:05.147761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.147749    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 04:05:05.147806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.147771    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 04:05:05.149740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.149720    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 04:05:05.149877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.149751    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 04:05:05.156218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156199    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 04:05:05.156387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156367    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 04:05:05.156514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156396    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 04:05:05.156637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156615    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 04:05:05.156693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156673    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 04:05:05.156940 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.156926    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 04:05:05.158050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158031    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 04:05:05.158050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158037    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 04:05:05.158209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158055    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 04:05:05.158209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158068    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 04:05:05.158209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158127    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 04:05:05.158209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.158131    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 04:05:05.159402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159381    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 04:05:05.159506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159422    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 04:05:05.159506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159460    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 04:05:05.159506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159424    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5b7hl\""
Oct 31 04:05:05.159506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159470    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 04:05:05.159647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159499    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 04:05:05.159647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159461    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 04:05:05.159789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159777    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 04:05:05.159846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159828    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 04:05:05.159885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159834    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 04:05:05.159885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159880    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-27zhj\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159840    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159853    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-76bmm1hqvtd1j\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159853    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ttr22\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159841    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ks42h\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159944    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 04:05:05.159958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159950    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lw8lb\""
Oct 31 04:05:05.160125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.159963    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 04:05:05.160202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.160170    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 04:05:05.160258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.160243    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bebll3gva5m81\""
Oct 31 04:05:05.160300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.160250    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 04:05:05.160300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.160273    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 04:05:05.186944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.186911    6931 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 04:05:05.190198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.190180    6931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:05.190299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.190285    6931 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:05.211203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.211151    6931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:05:05.211377 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.211223    6931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:05.215402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.211870    6931 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 04:05:05.215402 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.211926    6931 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Oct 31 04:05:05.230222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230194    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.230222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230223    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:05:05.230401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230243    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.230401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230259    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.230401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230276    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.230401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230297    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.230547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230423    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.230547 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.230467    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:05.230547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230529    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.230662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230590    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.230662 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.230607    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.730583953 +0000 UTC m=+1.763583624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:05.230662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230627    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.230662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230647    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230676    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230695    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230711    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230729    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-daemon-config\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230738    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.230855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230763    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230921    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230956    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230987    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230997    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.230997    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231014    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231042    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231066    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.231113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231091    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231195    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-tuned\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231205    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231256    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs52\" (UniqueName: \"kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231303    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca197fb2-de4b-4936-9af4-e1177179830c-ovn-node-metrics-cert\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231377    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231387    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-metrics-server-audit-profiles\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231409    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231440    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231473    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231516    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231552    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.231574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231579    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231580    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231615    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kmp\" (UniqueName: \"kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231630    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231645    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231669    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231694    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.231713    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.231720    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231727    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231719    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992wh\" (UniqueName: \"kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231776    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.231795    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.731778909 +0000 UTC m=+1.764778582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231811    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.231847    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.731830749 +0000 UTC m=+1.764830413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231876    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231906    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231932    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231949    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231953    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-script-lib\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231971    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.231996    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5w5\" (UniqueName: \"kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232003    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232015    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de8cf693-3022-4ad6-a50a-82466ad009ba-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232014    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/95ce4d22-fc05-4428-a9dd-651a3b1c4686-agent-certs\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232022    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232056    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232079    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232097    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbshb\" (UniqueName: \"kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232112    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232133    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.232149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232176    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232190    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e57f88b0-c977-4b88-82b3-14f7b7792845-iptables-alerter-script\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232204    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232223    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232221    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232261    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232279    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232368    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232410    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-grpc-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232437    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232461    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbvn\" (UniqueName: \"kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232477    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232478    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35f518cc-282f-4dc6-9279-4e162eba47ab-metrics-client-ca\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232497    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fbb956a-7849-497a-9463-f85599ef4d57-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232513    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232498    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-ovnkube-config\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232556    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232568    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232593    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232611    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232627    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232665    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232691    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjvx\" (UniqueName: \"kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232707    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232732    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232755    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232763    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232771    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232795    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232826    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d7df10-6133-426c-a8e1-48c78cdc1f44-tmp-dir\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232841    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232900    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232944    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232954    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.232983    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233008    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233046    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233084    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233143    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233173    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233208    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233237    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233265    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233294    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht87\" (UniqueName: \"kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233319    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233241    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233319    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233368    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-tls\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.233361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233383    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmff\" (UniqueName: \"kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233466    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233550    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233583    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233598    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6c8de0-72bd-40d8-aa86-34805cf518d0-audit-log\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233613    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233639    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233669    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233703    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233729    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233754    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233782    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233782    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233782    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233821    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233850    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233869    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-textfile\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233878    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233928    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.233940    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cni-binary-copy\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.234038    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234064    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234066    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.234093    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.734079264 +0000 UTC m=+1.767078922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234111    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234125    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234131    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234131    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7df10-6133-426c-a8e1-48c78cdc1f44-config-volume\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234192    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234287    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234328    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234340    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234349    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234370    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-telemeter-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234379    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234394    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f518cc-282f-4dc6-9279-4e162eba47ab-tmp\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234408    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234400    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e10b9e-7427-4d6d-9a99-e17824c49078-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234414    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234459    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234498    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234497    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234585    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-secret-metrics-server-client-certs\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234593    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234618    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234640    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234667    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234691    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234697    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234708    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234756    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234785    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234809    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234844    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234872    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt257\" (UniqueName: \"kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234897    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234926    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234948    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234969    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4502cc34-9b28-4f7e-a13b-594221372bbf-original-pull-secret\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.235035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.234975    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235173    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a345e1d5-f60a-473a-9cef-11db043e79df-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235229    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5460fbe1-498a-4aae-b32d-b8d97666d7b0-tmp\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235228    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235264    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/95ce4d22-fc05-4428-a9dd-651a3b1c4686-konnectivity-ca\") pod \"konnectivity-agent-gk4cs\" (UID: \"95ce4d22-fc05-4428-a9dd-651a3b1c4686\") " pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235299    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.235310    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235344    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8k98\" (UniqueName: \"kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235383    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c83f2509-20b7-4ece-9477-abeda7a82df1-cert\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235405    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a345e1d5-f60a-473a-9cef-11db043e79df-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.235418    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.735403346 +0000 UTC m=+1.768403020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235440    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235452    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235473    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235481    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235500    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235530    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235556    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235598    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsm8k\" (UniqueName: \"kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235630    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235636    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-data-volume\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235648    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-tls\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235658    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235661    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235770    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235788    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235805    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.235946    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-serving-certs-ca-bundle\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236043    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx49j\" (UniqueName: \"kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236073    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236071    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236115    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236138    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-federate-client-tls\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236147    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236174    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-98quaske5a7de\""
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236177    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236192    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4z92\" (UniqueName: \"kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236213    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236216    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236246    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236254    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acf269a0-e3d7-4932-ba07-f4f48ee0c787-tmp-dir\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236268    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236341    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c15711-14f6-494e-a518-478f433cfeb1-metrics-certs\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236374    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236403    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236408    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236428    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236452    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236477    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236508    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236628    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf9t\" (UniqueName: \"kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236629    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236691    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-tls\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236739    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236750    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5d7df10-6133-426c-a8e1-48c78cdc1f44-metrics-tls\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236767    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236774    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ca7768dc-fc1a-405d-92cc-75e17eaccf19-serviceca\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236788    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236808    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.236824    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236831    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt4q\" (UniqueName: \"kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.236854    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.736844768 +0000 UTC m=+1.769844424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236876    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236899    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236917    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236939    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236942    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.236987    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237014    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237040    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237072    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237100    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237125    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237135    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237127    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wdt\" (UniqueName: \"kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237125    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237185    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca197fb2-de4b-4936-9af4-e1177179830c-env-overrides\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237192    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237215    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237233    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237252    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237284    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nc9\" (UniqueName: \"kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237292    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-secret-telemeter-client\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237324    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237315    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f60643a3-3f4e-4308-a52e-0893b0fce7eb-metrics-client-ca\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237403    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6c8de0-72bd-40d8-aa86-34805cf518d0-client-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237416    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237446    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237480    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.237459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237506    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237533    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237541    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237559    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237584    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237609    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237634    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237658    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237682    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237685    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237717    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237744    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237850    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35f518cc-282f-4dc6-9279-4e162eba47ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237916    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.237960    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-metrics-client-ca\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.238030    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.239878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.238141    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6c8de0-72bd-40d8-aa86-34805cf518d0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.244832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.244815    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.245985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.245967    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.253590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.253573    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wsd\" (UniqueName: \"kubernetes.io/projected/88c15711-14f6-494e-a518-478f433cfeb1-kube-api-access-k6wsd\") pod \"network-metrics-daemon-dk458\" (UID: \"88c15711-14f6-494e-a518-478f433cfeb1\") " pod="openshift-multus/network-metrics-daemon-dk458"
Oct 31 04:05:05.254304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.254289    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs52\" (UniqueName: \"kubernetes.io/projected/b5d7df10-6133-426c-a8e1-48c78cdc1f44-kube-api-access-vrs52\") pod \"dns-default-v6nhz\" (UID: \"b5d7df10-6133-426c-a8e1-48c78cdc1f44\") " pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:05.255332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255077    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht87\" (UniqueName: \"kubernetes.io/projected/1fbb956a-7849-497a-9463-f85599ef4d57-kube-api-access-vht87\") pod \"monitoring-plugin-77f4f76594-brlvf\" (UID: \"1fbb956a-7849-497a-9463-f85599ef4d57\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:05:05.255332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255281    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kmp\" (UniqueName: \"kubernetes.io/projected/ca7768dc-fc1a-405d-92cc-75e17eaccf19-kube-api-access-c9kmp\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.255474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255377    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5w5\" (UniqueName: \"kubernetes.io/projected/acf269a0-e3d7-4932-ba07-f4f48ee0c787-kube-api-access-mn5w5\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.255474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255438    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmff\" (UniqueName: \"kubernetes.io/projected/eb6c8de0-72bd-40d8-aa86-34805cf518d0-kube-api-access-qlmff\") pod \"metrics-server-7b9ddc4b6b-k2l56\" (UID: \"eb6c8de0-72bd-40d8-aa86-34805cf518d0\") " pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:05.255587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255477    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjvx\" (UniqueName: \"kubernetes.io/projected/f60643a3-3f4e-4308-a52e-0893b0fce7eb-kube-api-access-sbjvx\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.255587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255550    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt257\" (UniqueName: \"kubernetes.io/projected/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kube-api-access-lt257\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.255718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.255700    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbshb\" (UniqueName: \"kubernetes.io/projected/de8cf693-3022-4ad6-a50a-82466ad009ba-kube-api-access-vbshb\") pod \"kube-state-metrics-5dc8d7d89b-vjhf4\" (UID: \"de8cf693-3022-4ad6-a50a-82466ad009ba\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjhf4"
Oct 31 04:05:05.260791 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.260773    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:05.260791 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.260791    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:05.260930 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.260800    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:05.260930 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.260849    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:05.760837806 +0000 UTC m=+1.793837491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:05.261299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.261276    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbvn\" (UniqueName: \"kubernetes.io/projected/f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a-kube-api-access-gwbvn\") pod \"telemeter-client-b8d6659cd-4n7qm\" (UID: \"f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a\") " pod="openshift-monitoring/telemeter-client-b8d6659cd-4n7qm"
Oct 31 04:05:05.261707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.261687    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8k98\" (UniqueName: \"kubernetes.io/projected/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-kube-api-access-j8k98\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.261863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.261845    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lg9\" (UniqueName: \"kubernetes.io/projected/ac673d9c-d013-487f-828d-72c57b8386b7-kube-api-access-c5lg9\") pod \"network-check-target-w49lj\" (UID: \"ac673d9c-d013-487f-828d-72c57b8386b7\") " pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:05:05.262680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262324    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4z92\" (UniqueName: \"kubernetes.io/projected/e57f88b0-c977-4b88-82b3-14f7b7792845-kube-api-access-p4z92\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.262680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262422    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsm8k\" (UniqueName: \"kubernetes.io/projected/5460fbe1-498a-4aae-b32d-b8d97666d7b0-kube-api-access-fsm8k\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.262680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262635    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nc9\" (UniqueName: \"kubernetes.io/projected/c83f2509-20b7-4ece-9477-abeda7a82df1-kube-api-access-89nc9\") pod \"ingress-canary-gqp52\" (UID: \"c83f2509-20b7-4ece-9477-abeda7a82df1\") " pod="openshift-ingress-canary/ingress-canary-gqp52"
Oct 31 04:05:05.262877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262686    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992wh\" (UniqueName: \"kubernetes.io/projected/ca197fb2-de4b-4936-9af4-e1177179830c-kube-api-access-992wh\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.262877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262710    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt4q\" (UniqueName: \"kubernetes.io/projected/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-kube-api-access-qkt4q\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.262877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262743    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf9t\" (UniqueName: \"kubernetes.io/projected/b6e10b9e-7427-4d6d-9a99-e17824c49078-kube-api-access-8sf9t\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.263041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.262924    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wdt\" (UniqueName: \"kubernetes.io/projected/a345e1d5-f60a-473a-9cef-11db043e79df-kube-api-access-z6wdt\") pod \"openshift-state-metrics-7bd7489957-nd47d\" (UID: \"a345e1d5-f60a-473a-9cef-11db043e79df\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nd47d"
Oct 31 04:05:05.263170 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.263139    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"prometheus-k8s-0\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:05.263230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.263143    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"alertmanager-main-0\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:05.263431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.263408    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx49j\" (UniqueName: \"kubernetes.io/projected/35f518cc-282f-4dc6-9279-4e162eba47ab-kube-api-access-gx49j\") pod \"thanos-querier-68c888f8cf-rcbzw\" (UID: \"35f518cc-282f-4dc6-9279-4e162eba47ab\") " pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:05.338301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338214    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338249    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.338301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338264    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338281    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338300    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338334    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-netns\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338346    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acf269a0-e3d7-4932-ba07-f4f48ee0c787-hosts-file\") pod \"node-resolver-phzfq\" (UID: \"acf269a0-e3d7-4932-ba07-f4f48ee0c787\") " pod="openshift-dns/node-resolver-phzfq"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338360    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-host\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338391    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338420    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-node-exporter-wtmp\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338422    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338450    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338452    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-conf\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338467    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-socket-dir-parent\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338477    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-node-log\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338498    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338505    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f79174e7-f9ee-4451-a10f-c8ffb4b4f948-crio-socket\") pod \"insights-runtime-extractor-jphhc\" (UID: \"f79174e7-f9ee-4451-a10f-c8ffb4b4f948\") " pod="openshift-insights/insights-runtime-extractor-jphhc"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338524    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338553    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338571    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338585    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.338584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338592    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-cnibin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338600    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338571    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysconfig\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338627    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-conf-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338637    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-modprobe-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338650    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338662    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-ovn\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338670    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338650    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-socket-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338681    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-kubelet\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338685    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338709    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-k8s-cni-cncf-io\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338720    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338727    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-multus-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338785    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338805    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-sysctl-d\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338805    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338823    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-kubelet\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338829    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338853    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338877    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338878    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-device-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338854    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-root\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338899    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338915    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338924    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e57f88b0-c977-4b88-82b3-14f7b7792845-host-slash\") pod \"iptables-alerter-tqzjj\" (UID: \"e57f88b0-c977-4b88-82b3-14f7b7792845\") " pod="openshift-network-operator/iptables-alerter-tqzjj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338928    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338963    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-system-cni-dir\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338983    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338999    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339014    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.338998    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-bin\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339020    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-systemd-units\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339015    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-registration-dir\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339029    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339043    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339049    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-kubernetes\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339056    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339061    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339077    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339084    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339091    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339088    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce70e1ef-553c-437b-bb60-f046bf05fa2e-sys-fs\") pod \"aws-ebs-csi-driver-node-24p8c\" (UID: \"ce70e1ef-553c-437b-bb60-f046bf05fa2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-24p8c"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339105    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-etc-kubernetes\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.339095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339111    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339123    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-etc-systemd\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339127    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339148    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339178    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-netns\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339198    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339207    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339225    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339248    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339282    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339296    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-netd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339297    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-run\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339306    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339317    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339341    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339357    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339361    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-run-systemd\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339382    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339395    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-dbus\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339397    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-var-lib-kubelet\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339401    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339419    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-log-socket\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339425    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4502cc34-9b28-4f7e-a13b-594221372bbf-kubelet-config\") pod \"global-pull-secret-syncer-s4d8g\" (UID: \"4502cc34-9b28-4f7e-a13b-594221372bbf\") " pod="kube-system/global-pull-secret-syncer-s4d8g"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339427    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339446    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339469    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339472    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-cni-bin\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339497    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339509    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-hostroot\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339512    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f60643a3-3f4e-4308-a52e-0893b0fce7eb-sys\") pod \"node-exporter-qfnsk\" (UID: \"f60643a3-3f4e-4308-a52e-0893b0fce7eb\") " pod="openshift-monitoring/node-exporter-qfnsk"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339521    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339531    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-host-slash\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339551    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-run-multus-certs\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339566    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339584    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339606    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339620    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339626    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-sys\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339637    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339651    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-system-cni-dir\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339662    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e10b9e-7427-4d6d-9a99-e17824c49078-cnibin\") pod \"multus-additional-cni-plugins-8rscm\" (UID: \"b6e10b9e-7427-4d6d-9a99-e17824c49078\") " pod="openshift-multus/multus-additional-cni-plugins-8rscm"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339664    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339684    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-var-lib-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339665    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19a1ddf5-4fd2-4b73-bf05-9e15e10aba57-host-var-lib-cni-multus\") pod \"multus-kgjcj\" (UID: \"19a1ddf5-4fd2-4b73-bf05-9e15e10aba57\") " pod="openshift-multus/multus-kgjcj"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339689    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339695    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca197fb2-de4b-4936-9af4-e1177179830c-etc-openvswitch\") pod \"ovnkube-node-vg88m\" (UID: \"ca197fb2-de4b-4936-9af4-e1177179830c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339714    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339752    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca7768dc-fc1a-405d-92cc-75e17eaccf19-host\") pod \"node-ca-xvz9v\" (UID: \"ca7768dc-fc1a-405d-92cc-75e17eaccf19\") " pod="openshift-image-registry/node-ca-xvz9v"
Oct 31 04:05:05.340364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.339757    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5460fbe1-498a-4aae-b32d-b8d97666d7b0-lib-modules\") pod \"tuned-rsdcv\" (UID: \"5460fbe1-498a-4aae-b32d-b8d97666d7b0\") " pod="openshift-cluster-node-tuning-operator/tuned-rsdcv"
Oct 31 04:05:05.744293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744201    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744259    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744286    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744314    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744340    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.744366    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744379    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744432    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744446    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744446    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744469    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:05.744506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744448    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.744432184 +0000 UTC m=+2.777431840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744384    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744535    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.744520491 +0000 UTC m=+2.777520155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744548    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.744541775 +0000 UTC m=+2.777541432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744562    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.74455566 +0000 UTC m=+2.777555317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744581    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.744568705 +0000 UTC m=+2.777568388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:05.744844 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.744600    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.744591101 +0000 UTC m=+2.777590773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:05.844854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:05.844815    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:05.845049 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.844998    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:05.845049 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.845024    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:05.845049 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.845037    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:05.845200 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:05.845105    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:06.845083115 +0000 UTC m=+2.878082790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:06.738801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.738769    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68c888f8cf-rcbzw"
Oct 31 04:05:06.755666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755625    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.755782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755682    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.755782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755723    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.755782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755745    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755798    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755749    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755806    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.755788529 +0000 UTC m=+4.788788186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755813    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755856    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.755844375 +0000 UTC m=+4.788844037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:06.755878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755852    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755884    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.755868527 +0000 UTC m=+4.788868217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755892    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755897    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755934    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.755923713 +0000 UTC m=+4.788923382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.755954    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.755944842 +0000 UTC m=+4.788944517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.755981    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.756038    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:06.756097 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.756065    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.756056778 +0000 UTC m=+4.789056469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:06.831570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.831544    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:06.856718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:06.856685    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:06.856872 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.856841    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:06.856872 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.856870    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:06.857046 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.856885    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:06.857046 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:06.856942    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:08.85692443 +0000 UTC m=+4.889924101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:07.291027 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:07.290987    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:05:07.555415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:07.555341    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v6nhz"
Oct 31 04:05:08.536725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.536698    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:08.776664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776632    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.776664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776669    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776721    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776758    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776758    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776793    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776815    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776834    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776814785 +0000 UTC m=+8.809814465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776854    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776858    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776849021 +0000 UTC m=+8.809848678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776854    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:08.776892 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776879    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.776900    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776938    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776944    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776913384 +0000 UTC m=+8.809913062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776972    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776962353 +0000 UTC m=+8.809962012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776986    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776980287 +0000 UTC m=+8.809979948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:08.777394 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.776997    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.776992195 +0000 UTC m=+8.809991856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:08.878639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:08.878521    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:08.878870 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.878815    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:08.878870 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.878843    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:08.878870 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.878856    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:08.878982 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:08.878925    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:12.87890727 +0000 UTC m=+8.911906929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:09.741857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:09.741821    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-brlvf"
Oct 31 04:05:10.757626 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:10.757591    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:05:10.964141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:10.964110    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:10.964389 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:10.964376    6931 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 04:05:12.821303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821261    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821306    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821347    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821378    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821389    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821419    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821442    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821489    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821488    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821449    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821430201 +0000 UTC m=+16.854429878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821492    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821609    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821587741 +0000 UTC m=+16.854587401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821634    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821628144 +0000 UTC m=+16.854627801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821649    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821642207 +0000 UTC m=+16.854641863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821670    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821660977 +0000 UTC m=+16.854660642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.821703    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.821782 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821782    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:12.822261 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.821822    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.821815892 +0000 UTC m=+16.854815553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:12.922374 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:12.922329    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:12.922583 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.922525    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:12.922583 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.922553    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:12.922583 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.922566    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:12.922733 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:12.922635    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:20.92261559 +0000 UTC m=+16.955615262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:13.734038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:13.733995    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:13.940392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:13.940356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:14.920307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:14.920222    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gk4cs"
Oct 31 04:05:20.584288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.584259    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b9ddc4b6b-k2l56"
Oct 31 04:05:20.906673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906596    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906645    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906675    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906703    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906718    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906723    6931 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:20.906753    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906797    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.906772179 +0000 UTC m=+32.939771837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"service-ca" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906826    6931 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906846    6931 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906896    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.906881664 +0000 UTC m=+32.939881333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-serving-cert" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906829    6931 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:20.906950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906923    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.906903689 +0000 UTC m=+32.939903366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-config" not registered
Oct 31 04:05:20.907313 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906832    6931 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:20.907313 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.906834    6931 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:20.907313 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.907022    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.907000438 +0000 UTC m=+32.940000099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 31 04:05:20.907313 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.907039    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.907030151 +0000 UTC m=+32.940029815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 31 04:05:20.907313 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:20.907053    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:36.907045511 +0000 UTC m=+32.940045169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : object "openshift-console"/"console-oauth-config" not registered
Oct 31 04:05:21.007739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:21.007706    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"console-8594bc5d5f-sn56k\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") " pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:21.007908 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:21.007878    6931 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 31 04:05:21.007908 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:21.007900    6931 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 31 04:05:21.007987 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:21.007911    6931 projected.go:196] Error preparing data for projected volume kube-api-access-d2vrn for pod openshift-console/console-8594bc5d5f-sn56k: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:21.007987 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:21.007965    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn podName:074b81bc-3e3f-4fc3-b6af-8ade5ea6673d nodeName:}" failed. No retries permitted until 2026-10-31 04:05:37.007951825 +0000 UTC m=+33.040951482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d2vrn" (UniqueName: "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn") pod "console-8594bc5d5f-sn56k" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 31 04:05:21.061175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:21.061135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:21.271955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:21.271874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:23.488978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:23.488936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:23.699521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:23.699486    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:23.908956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:23.908917    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:26.031401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.031367    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:26.032351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.032299    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="alertmanager" containerID="cri-o://285d8109a3ac4b8d5a9369971a58f7972d71c9cc5862ca4350835cc91c7081f9" gracePeriod=120
Oct 31 04:05:26.032868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.032841    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="prom-label-proxy" containerID="cri-o://1ac0225044e6ff6de1bfff7985284a23742883b88d4b966714ac08b1765e47fd" gracePeriod=120
Oct 31 04:05:26.033065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.033048    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-metric" containerID="cri-o://5dce22cace3775452903aab027722ae61094dadea523e6371016b7871438707c" gracePeriod=120
Oct 31 04:05:26.033269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.033254    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy" containerID="cri-o://15a28ac42fa5b0e9dea504db790a499f8ab8576749caf0dc77f1346d991cb6aa" gracePeriod=120
Oct 31 04:05:26.034890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.034373    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-web" containerID="cri-o://8afcd2d7211725b74f44efe040584c3b39449208d0c2ef1f6e98101551db0628" gracePeriod=120
Oct 31 04:05:26.034998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.034412    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="config-reloader" containerID="cri-o://4f0a53c703af8e5bedcf012c45e85e9f67290e96efc17724250ce645533e4f25" gracePeriod=120
Oct 31 04:05:26.273563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273527    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="1ac0225044e6ff6de1bfff7985284a23742883b88d4b966714ac08b1765e47fd" exitCode=0
Oct 31 04:05:26.273563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273550    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="5dce22cace3775452903aab027722ae61094dadea523e6371016b7871438707c" exitCode=0
Oct 31 04:05:26.273563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273556    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="15a28ac42fa5b0e9dea504db790a499f8ab8576749caf0dc77f1346d991cb6aa" exitCode=0
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273582    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="4f0a53c703af8e5bedcf012c45e85e9f67290e96efc17724250ce645533e4f25" exitCode=0
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273589    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="285d8109a3ac4b8d5a9369971a58f7972d71c9cc5862ca4350835cc91c7081f9" exitCode=0
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273603    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"1ac0225044e6ff6de1bfff7985284a23742883b88d4b966714ac08b1765e47fd"}
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273673    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"5dce22cace3775452903aab027722ae61094dadea523e6371016b7871438707c"}
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273700    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"15a28ac42fa5b0e9dea504db790a499f8ab8576749caf0dc77f1346d991cb6aa"}
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273724    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"4f0a53c703af8e5bedcf012c45e85e9f67290e96efc17724250ce645533e4f25"}
Oct 31 04:05:26.273786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:26.273747    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"285d8109a3ac4b8d5a9369971a58f7972d71c9cc5862ca4350835cc91c7081f9"}
Oct 31 04:05:27.280524 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.280492    6931 generic.go:356] "Generic (PLEG): container finished" podID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerID="8afcd2d7211725b74f44efe040584c3b39449208d0c2ef1f6e98101551db0628" exitCode=0
Oct 31 04:05:27.280868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.280571    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"8afcd2d7211725b74f44efe040584c3b39449208d0c2ef1f6e98101551db0628"}
Oct 31 04:05:27.280868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.280630    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"337b4d1b-8c2b-4ba1-a463-38484614334c","Type":"ContainerDied","Data":"ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5"}
Oct 31 04:05:27.280868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.280656    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad700e398906461d71f980d0a536ca756a15b9c7ba7e57dcfe86cadc1aabaab5"
Oct 31 04:05:27.281874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.281858    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.379891    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.379972    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380006    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380033    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380101    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380130    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380187    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380221    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380276    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380314    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380351    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380380    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.380772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.380419    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") pod \"337b4d1b-8c2b-4ba1-a463-38484614334c\" (UID: \"337b4d1b-8c2b-4ba1-a463-38484614334c\") "
Oct 31 04:05:27.381910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.381568    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:27.381910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.381779    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:27.383091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.382814    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:27.385550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.385519    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:27.385888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.385857    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.386899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.386867    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.387087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.386973    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.389008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.388969    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.389385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.389351    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.389631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.389609    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "kube-api-access-nzkxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:27.390748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.390723    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.393468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.393445    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:27.399729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.399687    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config" pod "337b4d1b-8c2b-4ba1-a463-38484614334c" (UID: "337b4d1b-8c2b-4ba1-a463-38484614334c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.481311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481262    6931 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-tls-assets\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481303    6931 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-main-db\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481317    6931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481328    6931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481339    6931 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481348    6931 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/337b4d1b-8c2b-4ba1-a463-38484614334c-config-out\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481356    6931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481365    6931 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481374    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/337b4d1b-8c2b-4ba1-a463-38484614334c-metrics-client-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481382    6931 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-web-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481391    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkxm\" (UniqueName: \"kubernetes.io/projected/337b4d1b-8c2b-4ba1-a463-38484614334c-kube-api-access-nzkxm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481398    6931 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-cluster-tls-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:27.481577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:27.481407    6931 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/337b4d1b-8c2b-4ba1-a463-38484614334c-secret-alertmanager-main-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:28.284674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.284634    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.348774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.348739    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:28.360454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.360387    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:28.385146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385115    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:28.385428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385414    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-metric"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385428    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-metric"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385448    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="prom-label-proxy"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385462    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="prom-label-proxy"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385470    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="alertmanager"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385475    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="alertmanager"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385483    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="init-config-reloader"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385487    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="init-config-reloader"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385498    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy"
Oct 31 04:05:28.385501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385503    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385515    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-web"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385520    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-web"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385525    6931 cpu_manager.go:414] "containerMap: removing container" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="config-reloader"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385530    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="config-reloader"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385569    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385578    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-web"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385583    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="config-reloader"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385591    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="kube-rbac-proxy-metric"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385602    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="prom-label-proxy"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385607    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="alertmanager"
Oct 31 04:05:28.385854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.385613    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" containerName="init-config-reloader"
Oct 31 04:05:28.393912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.393883    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.409149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409115    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 04:05:28.409570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409541    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 04:05:28.409645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409615    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 04:05:28.409684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409665    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 04:05:28.409684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409669    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 04:05:28.409757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409695    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 04:05:28.409757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409702    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-87lk2\""
Oct 31 04:05:28.409757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409735    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 04:05:28.409757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.409746    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 04:05:28.421191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.421136    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 04:05:28.421850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.421829    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:28.492611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492580    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-web-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492614    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-out\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492749    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492771    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492794    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492940 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492825    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.492940 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492870    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.492977    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llmpj\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-kube-api-access-llmpj\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.493040    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.493110    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.493135    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.493234    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.493283 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.493258    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-volume\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593643    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593688    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593855    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593896    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-volume\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593930    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-web-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.593997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.593962    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-out\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594052    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594090    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594250    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594289    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594329    6931 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594343    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594371    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594405    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llmpj\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-kube-api-access-llmpj\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594444    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.594899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.594572    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.595185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.595143    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.597040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.597010    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-out\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.597461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.597420    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.597554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.597535    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.597614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.597574    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.597671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.597613    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-config-volume\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.598108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.598065    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-web-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.598214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.598176    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.598445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.598419    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.598562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.598546    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.617701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.617638    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llmpj\" (UniqueName: \"kubernetes.io/projected/31fdcc1c-2f57-4ded-bbe7-1d07aea75705-kube-api-access-llmpj\") pod \"alertmanager-main-0\" (UID: \"31fdcc1c-2f57-4ded-bbe7-1d07aea75705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.662637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.662599    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:28.715315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.715278    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 04:05:28.874016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:28.873930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:29.050294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:29.050265    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 04:05:29.055267 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:29.055239    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fdcc1c_2f57_4ded_bbe7_1d07aea75705.slice/crio-8c8104b30dd83cbb5046ebbb279010dff7347b7317238e57229ec9db6db2530f WatchSource:0}: Error finding container 8c8104b30dd83cbb5046ebbb279010dff7347b7317238e57229ec9db6db2530f: Status 404 returned error can't find the container with id 8c8104b30dd83cbb5046ebbb279010dff7347b7317238e57229ec9db6db2530f
Oct 31 04:05:29.290501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:29.290465    6931 generic.go:356] "Generic (PLEG): container finished" podID="31fdcc1c-2f57-4ded-bbe7-1d07aea75705" containerID="8525216c09eedeff2967a8f7e91c25915d6f14974406cfd0d0b7337d58ae5756" exitCode=0
Oct 31 04:05:29.290879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:29.290553    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerDied","Data":"8525216c09eedeff2967a8f7e91c25915d6f14974406cfd0d0b7337d58ae5756"}
Oct 31 04:05:29.290879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:29.290629    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"8c8104b30dd83cbb5046ebbb279010dff7347b7317238e57229ec9db6db2530f"}
Oct 31 04:05:30.122888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.122854    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337b4d1b-8c2b-4ba1-a463-38484614334c" path="/var/lib/kubelet/pods/337b4d1b-8c2b-4ba1-a463-38484614334c/volumes"
Oct 31 04:05:30.153657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.153599    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8594bc5d5f-sn56k" podUID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" containerName="console" containerID="cri-o://5169afc028c4bfe4de6e4f062faeb5a4d9d4858d30a6c7c6cf9f02d2633d8482" gracePeriod=15
Oct 31 04:05:30.298448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298407    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"4bd4a319ebe0b7976eee94bf0f2eac5d298dc715b428f188932418ef45c0c6d2"}
Oct 31 04:05:30.298948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298499    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"d709c29e922bcf86fd5cc1f197ad4225a04b182be8732cd18392d11ab4c08a60"}
Oct 31 04:05:30.298948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298543    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"3d5a463cbf56635e30f14e87c5f58b0a73fe4b5f714d769df21aab190f115e7e"}
Oct 31 04:05:30.298948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298582    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"83d63721b3dc8a938180bcc851ff73c93ab2295c3d6f510818603449d16af213"}
Oct 31 04:05:30.298948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298631    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"89451170bc3d5afbdb283802ca3df1c8c1f53906cdba05a3000f7f608880d5de"}
Oct 31 04:05:30.298948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.298667    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31fdcc1c-2f57-4ded-bbe7-1d07aea75705","Type":"ContainerStarted","Data":"61b82db3431aabca8b017a01e258cac160980e87812ac72cb321288fe3b7b6cb"}
Oct 31 04:05:30.299998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.299974    6931 generic.go:356] "Generic (PLEG): container finished" podID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" containerID="5169afc028c4bfe4de6e4f062faeb5a4d9d4858d30a6c7c6cf9f02d2633d8482" exitCode=0
Oct 31 04:05:30.300090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.300007    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8594bc5d5f-sn56k" event={"ID":"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d","Type":"ContainerDied","Data":"5169afc028c4bfe4de6e4f062faeb5a4d9d4858d30a6c7c6cf9f02d2633d8482"}
Oct 31 04:05:30.362738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.362679    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.352895592 podStartE2EDuration="2.362666609s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.771017ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:30.361944033 +0000 UTC m=+26.394943769" watchObservedRunningTime="2026-10-31 04:05:30.362666609 +0000 UTC m=+26.395666278"
Oct 31 04:05:30.390294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.390267    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:30.503010 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.502981    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vg88m"
Oct 31 04:05:30.512199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512171    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512220    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512252    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512279    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512317    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512399    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512470    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512507    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") pod \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\" (UID: \"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d\") "
Oct 31 04:05:30.512721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512691    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:30.512721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512699    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:30.512840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512801    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:30.512877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.512840    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:30.513092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.513068    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:30.514514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.514492    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "kube-api-access-d2vrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:30.514624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.514537    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:30.514624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.514603    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert" pod "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" (UID: "074b81bc-3e3f-4fc3-b6af-8ade5ea6673d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:30.614208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614138    6931 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-oauth-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614201    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vrn\" (UniqueName: \"kubernetes.io/projected/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-kube-api-access-d2vrn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614219    6931 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-tmp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614235    6931 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614249    6931 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614261    6931 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-oauth-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614273    6931 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.614483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.614287    6931 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d-console-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:30.667086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.666991    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667359    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy" containerID="cri-o://40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327" gracePeriod=600
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667380    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="prometheus" containerID="cri-o://de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85" gracePeriod=600
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667393    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-thanos" containerID="cri-o://712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d" gracePeriod=600
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667429    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="config-reloader" containerID="cri-o://3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919" gracePeriod=600
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667403    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-web" containerID="cri-o://c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f" gracePeriod=600
Oct 31 04:05:30.667535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.667393    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="thanos-sidecar" containerID="cri-o://ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce" gracePeriod=600
Oct 31 04:05:30.904288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:30.904264    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.016903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016820    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.016903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016880    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016902    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016938    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016957    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.016983    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017018    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017072    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017099    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017141    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017179    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017243    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017276    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017311    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017342    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017390    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017421    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017453    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017482    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017513    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") pod \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\" (UID: \"2fb56708-d5b3-47bc-9bb6-1d931829bc34\") "
Oct 31 04:05:31.017911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017551    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.017911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.017763    6931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.018829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.018407    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.018829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.018540    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.019325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.019298    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.019594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.019570    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.019825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.019799    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:31.020092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.020068    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.020209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.020186    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.021012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.020988    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:31.021895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.021874    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.022049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022032    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "kube-api-access-fhfhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:31.022127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022059    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:31.022127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022105    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.022294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022276    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.022505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022457    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.022615 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022508    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.022965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.022945    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:31.023131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.023114    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.023344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.023322    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.032819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.032800    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config" pod "2fb56708-d5b3-47bc-9bb6-1d931829bc34" (UID: "2fb56708-d5b3-47bc-9bb6-1d931829bc34"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:31.118962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.118934    6931 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-metrics-client-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.118962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.118964    6931 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.118974    6931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.118985    6931 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.118996    6931 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-web-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119005    6931 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-metrics-client-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119016    6931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119025    6931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119034    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhfhh\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-kube-api-access-fhfhh\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119042    6931 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119052    6931 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-config-out\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119060    6931 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-kube-rbac-proxy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119068    6931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-db\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119076    6931 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119088    6931 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119102    6931 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2fb56708-d5b3-47bc-9bb6-1d931829bc34-tls-assets\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119115    6931 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119126    6931 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb56708-d5b3-47bc-9bb6-1d931829bc34-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.119135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.119140    6931 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2fb56708-d5b3-47bc-9bb6-1d931829bc34-secret-grpc-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:05:31.307741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307699    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d" exitCode=0
Oct 31 04:05:31.307741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307732    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327" exitCode=0
Oct 31 04:05:31.307741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307742    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f" exitCode=0
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307751    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce" exitCode=0
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307760    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919" exitCode=0
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307771    6931 generic.go:356] "Generic (PLEG): container finished" podID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85" exitCode=0
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307778    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307801    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307843    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307878    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307906    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307914    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307937    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307970    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"}
Oct 31 04:05:31.308252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.307995    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2fb56708-d5b3-47bc-9bb6-1d931829bc34","Type":"ContainerDied","Data":"cb8c3529edab7c5bf887db40b4370db3f165df3102f0c06c0f8df90cda538c9b"}
Oct 31 04:05:31.315182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.312671    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8594bc5d5f-sn56k"
Oct 31 04:05:31.315182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.312694    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8594bc5d5f-sn56k" event={"ID":"074b81bc-3e3f-4fc3-b6af-8ade5ea6673d","Type":"ContainerDied","Data":"1cfd2f007cc933816ff52259a7ba707291a78679159560d8aed8d928648ff153"}
Oct 31 04:05:31.320619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.320593    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.331787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.331768    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.338582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.338555    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.345044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.345027    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.351895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.351877    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.359108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.359090    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.359278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.359259    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:05:31.365259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.365237    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8594bc5d5f-sn56k"]
Oct 31 04:05:31.366540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.366520    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.366789 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.366769    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.366842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.366798    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.366842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.366829    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.367054 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.367040    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.367092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367059    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.367092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367071    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.367308 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.367291    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.367353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367315    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.367353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367330    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.367562 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.367548    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.367600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367568    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.367600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367580    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.367789 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.367774    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.367830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367792    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.367830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367804    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.367993 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.367979    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.368040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.367996    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.368040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368007    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.368466 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:31.368452    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.368500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368469    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.368500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368483    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.368683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368665    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.368730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368684    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.368900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368885    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.368942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.368900    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.369111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369087    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.369146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369113    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.369375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369358    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.369418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369375    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.369580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369564    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.369614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369579    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.369815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369799    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.369856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369815    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.370005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.369989    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.370043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370005    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.370213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370198    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.370256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370213    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.370413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370394    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.370467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370414    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.370631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370616    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.370674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370631    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.370852 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370829    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.370852 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.370851    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.371051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371035    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.371051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371051    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.371262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371243    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.371302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371262    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.371464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371437    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.371502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371465    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.371660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371641    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.371737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371662    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.371872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371853    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.371872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.371871    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.372055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372037    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.372097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372056    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.372263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372245    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.372314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372263    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.372450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372434    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.372497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372451    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.372670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372655    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.372708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372670    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.372869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372854    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.372917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.372869    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.373059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373041    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.373095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373059    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.373273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373257    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.373357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373272    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.373474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373457    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.373518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373475    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.373666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373651    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.373701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373665    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.373833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373817    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.373871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.373833    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.374029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374014    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.374076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374029    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.374232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374217    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.374281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374232    6931 scope.go:122] "RemoveContainer" containerID="712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"
Oct 31 04:05:31.374423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374406    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d"} err="failed to get container status \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": rpc error: code = NotFound desc = could not find container \"712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d\": container with ID starting with 712eec4dbb28b39f6eccf82e2104a173d2cf115640201635ee91c84eeb6b138d not found: ID does not exist"
Oct 31 04:05:31.374423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374422    6931 scope.go:122] "RemoveContainer" containerID="40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"
Oct 31 04:05:31.374614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374599    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327"} err="failed to get container status \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": rpc error: code = NotFound desc = could not find container \"40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327\": container with ID starting with 40ee40cd4d1c296e3a8b382abad377f2f4c436615a600e6b29471ce5cf5da327 not found: ID does not exist"
Oct 31 04:05:31.374651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374614    6931 scope.go:122] "RemoveContainer" containerID="c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"
Oct 31 04:05:31.374805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374780    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f"} err="failed to get container status \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": rpc error: code = NotFound desc = could not find container \"c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f\": container with ID starting with c5c789ed9fae00212f7fc85e19dd67f66d803cb1b98089addc662fe90aed3d6f not found: ID does not exist"
Oct 31 04:05:31.374805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.374802    6931 scope.go:122] "RemoveContainer" containerID="ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"
Oct 31 04:05:31.375036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375020    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce"} err="failed to get container status \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": rpc error: code = NotFound desc = could not find container \"ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce\": container with ID starting with ea1a13ddf9ab9d8ac0cf1d536e53abdaa7a64eff8610975ab7820545dcbfacce not found: ID does not exist"
Oct 31 04:05:31.375036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375036    6931 scope.go:122] "RemoveContainer" containerID="3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"
Oct 31 04:05:31.375229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375214    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919"} err="failed to get container status \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": rpc error: code = NotFound desc = could not find container \"3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919\": container with ID starting with 3ada29704caafc09e2f32871c017446bd0f95f9814183820230fcdf6564c9919 not found: ID does not exist"
Oct 31 04:05:31.375272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375229    6931 scope.go:122] "RemoveContainer" containerID="de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"
Oct 31 04:05:31.375412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375384    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85"} err="failed to get container status \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": rpc error: code = NotFound desc = could not find container \"de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85\": container with ID starting with de5cfac7ee1895b34b15d7f5223b5152cde653646727ac860eab11c72bd3ce85 not found: ID does not exist"
Oct 31 04:05:31.375463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375412    6931 scope.go:122] "RemoveContainer" containerID="a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"
Oct 31 04:05:31.375670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375654    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b"} err="failed to get container status \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": rpc error: code = NotFound desc = could not find container \"a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b\": container with ID starting with a6d0954c6f8e26ee836ba994d4effeea09951c8fe18691ed3666b053944c3e5b not found: ID does not exist"
Oct 31 04:05:31.375670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.375668    6931 scope.go:122] "RemoveContainer" containerID="5169afc028c4bfe4de6e4f062faeb5a4d9d4858d30a6c7c6cf9f02d2633d8482"
Oct 31 04:05:31.399975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.399951    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:31.422481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.422453    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:31.439133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439109    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:31.439438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439425    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="thanos-sidecar"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439439    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="thanos-sidecar"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439451    6931 cpu_manager.go:414] "containerMap: removing container" podUID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" containerName="console"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439457    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" containerName="console"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439468    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="prometheus"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439473    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="prometheus"
Oct 31 04:05:31.439488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439487    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-web"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439492    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-web"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439498    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439502    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439509    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="config-reloader"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439514    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="config-reloader"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439523    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-thanos"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439527    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-thanos"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439537    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="init-config-reloader"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439541    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="init-config-reloader"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439585    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="thanos-sidecar"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439592    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" containerName="console"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439599    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="prometheus"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439606    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-web"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439611    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439616    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="config-reloader"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439622    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="kube-rbac-proxy-thanos"
Oct 31 04:05:31.439654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.439628    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" containerName="init-config-reloader"
Oct 31 04:05:31.443602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.443584    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.458267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.458241    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 04:05:31.458267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.458241    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 04:05:31.458474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.458308    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 04:05:31.458956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.458939    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 04:05:31.459026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.458983    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 04:05:31.470207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470184    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 04:05:31.470207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470202    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 04:05:31.470366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470218    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 04:05:31.470366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470334    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 04:05:31.470366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470334    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 04:05:31.470516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470396    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lw8lb\""
Oct 31 04:05:31.470516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470410    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 04:05:31.470516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470415    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 04:05:31.470685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.470671    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-98quaske5a7de\""
Oct 31 04:05:31.471320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.471294    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 04:05:31.471456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.471420    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 04:05:31.472431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.472411    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:31.523069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523036    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523094    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523197    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523238    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523283    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523326    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523355    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523379    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523421    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523458    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523481    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523583    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523613    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523642    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523665    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523704    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523729    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwn4\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-kube-api-access-tdwn4\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523751    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523775    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.523984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.523857    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.624724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624634    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.624724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624672    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.624724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624697    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.624724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624720    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624750    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624773    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwn4\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-kube-api-access-tdwn4\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624794    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624817    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624850    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624875    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624925    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624947    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.624978    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625010    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625038    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625074    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625104    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625127    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625190    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625224    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625285    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625552    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.625945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.625883    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.627075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.626057    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.627075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.626757    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.627075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.626816    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.627727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.627140    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.628091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.628020    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.628956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.628400    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.628956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.628873    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.628956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.628906    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.629210 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.629185    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.629302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.629281    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.629722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.629698    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.630201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.630181    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.630379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.630362    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.630592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.630575    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.630722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.630704    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.630950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.630930    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.648324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.648300    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwn4\" (UniqueName: \"kubernetes.io/projected/5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3-kube-api-access-tdwn4\") pod \"prometheus-k8s-0\" (UID: \"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:31.761405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:31.761375    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:32.097250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.097212    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 04:05:32.102128 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:05:32.102100    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b71d59e_2ca7_49a1_b07b_fb44ad47dbc3.slice/crio-ef32a341055d66c71494b44224ef1e29d4cf5a4322198287f823370f42314891 WatchSource:0}: Error finding container ef32a341055d66c71494b44224ef1e29d4cf5a4322198287f823370f42314891: Status 404 returned error can't find the container with id ef32a341055d66c71494b44224ef1e29d4cf5a4322198287f823370f42314891
Oct 31 04:05:32.125461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.121277    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074b81bc-3e3f-4fc3-b6af-8ade5ea6673d" path="/var/lib/kubelet/pods/074b81bc-3e3f-4fc3-b6af-8ade5ea6673d/volumes"
Oct 31 04:05:32.125461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.121881    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb56708-d5b3-47bc-9bb6-1d931829bc34" path="/var/lib/kubelet/pods/2fb56708-d5b3-47bc-9bb6-1d931829bc34/volumes"
Oct 31 04:05:32.320293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.320264    6931 generic.go:356] "Generic (PLEG): container finished" podID="5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3" containerID="a35d64e94003f57eef3fd4b0f54b579477945b9dcd3c989976bdb14fc4b859e9" exitCode=0
Oct 31 04:05:32.320714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.320306    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerDied","Data":"a35d64e94003f57eef3fd4b0f54b579477945b9dcd3c989976bdb14fc4b859e9"}
Oct 31 04:05:32.320714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:32.320361    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"ef32a341055d66c71494b44224ef1e29d4cf5a4322198287f823370f42314891"}
Oct 31 04:05:33.328151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328116    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"432caa5d752aa155307e860778a9683131883a1512e482eaf59c04698fa50bfa"}
Oct 31 04:05:33.328559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328194    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"6385ca71c3c3b6ec0bcc63b691d45d839e3cac12db61064a0f88ac402c21d2da"}
Oct 31 04:05:33.328559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328221    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"58cbfc6030244a9c2f4872b58c97f9d0b85c9b1f6d4964aeb076d25d53a81ff2"}
Oct 31 04:05:33.328559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328248    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"8a36f4fb119c1676721c52f79c9793714b09d09361cae1ce37493885c2cd1b85"}
Oct 31 04:05:33.328559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328274    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"06a648955dd58bb8c2ecf6ad08bb03f3be2f0bebeca8079c15118e429e29951b"}
Oct 31 04:05:33.328559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.328301    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3","Type":"ContainerStarted","Data":"9e55542b4db852f489c8595c4acd78de058bc9f2861fdde2174de4bfdc414733"}
Oct 31 04:05:33.394429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:33.394375    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.378618944 podStartE2EDuration="2.394358501s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.739557ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:33.393617057 +0000 UTC m=+29.426616735" watchObservedRunningTime="2026-10-31 04:05:33.394358501 +0000 UTC m=+29.427358218"
Oct 31 04:05:36.762375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:36.762335    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:37.293380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:37.293349    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w49lj"
Oct 31 04:05:40.758092 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:05:40.758037    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:05:43.671133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:43.671096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:43.882088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:43.882052    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:54.519039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:54.519002    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:59.818674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:59.818636    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:05:59.835009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:05:59.834980    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:06:00.449659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:00.449628    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 04:06:02.389107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:02.389072    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:10.757456 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:06:10.757417    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:06:16.168571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:16.168529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:40.758243 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:06:40.758194    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:06:43.120289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:43.120251    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:43.336179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:43.336131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:54.642092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:06:54.642005    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:10.757451 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:07:10.757408    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:07:40.758183 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:07:40.758132    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:07:54.763627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:07:54.763591    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:10.758240 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:08:10.758202    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:08:15.895903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:15.895874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:16.140026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:16.139981    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:16.694929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:16.694895    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:17.507147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:17.507104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:17.737072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:17.737028    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:18.781297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:18.781259    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:21.801753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:21.801719    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:22.018765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:22.018726    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:22.933440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:22.933404    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:23.194207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.194087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:23.240800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.240768    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"]
Oct 31 04:08:23.243343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.243324    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.269210 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.269184    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 04:08:23.269378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.269333    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-tp9sg\""
Oct 31 04:08:23.269453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.269389    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 04:08:23.270054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.270033    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"]
Oct 31 04:08:23.298617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.298589    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.298792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.298625    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkz4\" (UniqueName: \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.298792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.298647    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.400131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.400088    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.400323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.400138    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkz4\" (UniqueName: \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.400323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.400181    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.400521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.400498    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.400615 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.400545    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.405452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.405425    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:23.423441 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.423412    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkz4\" (UniqueName: \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.555895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.555864    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:23.771845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.771802    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"]
Oct 31 04:08:23.774464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.774443    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.788746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.788722    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-zvgb6\""
Oct 31 04:08:23.795208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.795182    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"]
Oct 31 04:08:23.803526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.803499    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.803664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.803544    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.803664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.803593    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp58q\" (UniqueName: \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.871386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.871303    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"]
Oct 31 04:08:23.876835 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:23.876809    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d800e6c_4e04_4b08_a196_6522516d4214.slice/crio-ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2 WatchSource:0}: Error finding container ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2: Status 404 returned error can't find the container with id ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2
Oct 31 04:08:23.878845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.878826    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:08:23.904548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.904517    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.904670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.904557    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.904670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.904584    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp58q\" (UniqueName: \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.904970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.904947    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.905029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.904969    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.927784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.927758    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp58q\" (UniqueName: \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:23.964059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:23.964029    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt" event={"ID":"0d800e6c-4e04-4b08-a196-6522516d4214","Type":"ContainerStarted","Data":"ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2"}
Oct 31 04:08:24.092117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:24.092076    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:24.394801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:24.394724    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"]
Oct 31 04:08:24.400635 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:24.400601    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d5b4fc_a21c_409e_bb47_1c81139b8639.slice/crio-0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f WatchSource:0}: Error finding container 0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f: Status 404 returned error can't find the container with id 0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f
Oct 31 04:08:24.970058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:24.970014    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r" event={"ID":"15d5b4fc-a21c-409e-bb47-1c81139b8639","Type":"ContainerStarted","Data":"0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f"}
Oct 31 04:08:26.143987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:26.143889    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:30.003598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:30.003561    6931 generic.go:356] "Generic (PLEG): container finished" podID="0d800e6c-4e04-4b08-a196-6522516d4214" containerID="75e85a92514056ca62fa757b5a2146373e35d0d5ac26f6192eceb0bbe954e90c" exitCode=0
Oct 31 04:08:30.004080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:30.003608    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt" event={"ID":"0d800e6c-4e04-4b08-a196-6522516d4214","Type":"ContainerDied","Data":"75e85a92514056ca62fa757b5a2146373e35d0d5ac26f6192eceb0bbe954e90c"}
Oct 31 04:08:31.010670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:31.010630    6931 generic.go:356] "Generic (PLEG): container finished" podID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerID="641f100fff3277be80aea0ed7ff4d03ae55e8b284e889562f16b5905ea98c6dd" exitCode=0
Oct 31 04:08:31.011189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:31.010717    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r" event={"ID":"15d5b4fc-a21c-409e-bb47-1c81139b8639","Type":"ContainerDied","Data":"641f100fff3277be80aea0ed7ff4d03ae55e8b284e889562f16b5905ea98c6dd"}
Oct 31 04:08:32.017489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:32.017403    6931 generic.go:356] "Generic (PLEG): container finished" podID="0d800e6c-4e04-4b08-a196-6522516d4214" containerID="505adfe0ae31f720f3fa1e2de3f2c74d7241561cf09410537a534ddd061bcd4d" exitCode=0
Oct 31 04:08:32.017933 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:32.017490    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt" event={"ID":"0d800e6c-4e04-4b08-a196-6522516d4214","Type":"ContainerDied","Data":"505adfe0ae31f720f3fa1e2de3f2c74d7241561cf09410537a534ddd061bcd4d"}
Oct 31 04:08:32.024865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:32.024836    6931 generic.go:356] "Generic (PLEG): container finished" podID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerID="161d2416aa4bb42ab2d6b5b5a34fec20eb6d5d147689c84b4862bae288e2cb6a" exitCode=0
Oct 31 04:08:32.024968 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:32.024915    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r" event={"ID":"15d5b4fc-a21c-409e-bb47-1c81139b8639","Type":"ContainerDied","Data":"161d2416aa4bb42ab2d6b5b5a34fec20eb6d5d147689c84b4862bae288e2cb6a"}
Oct 31 04:08:39.061780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:39.061744    6931 generic.go:356] "Generic (PLEG): container finished" podID="0d800e6c-4e04-4b08-a196-6522516d4214" containerID="1f0745299458b6bdf0d8163e69f0779838df766c4c08b5def42fb1ef5339cbe6" exitCode=0
Oct 31 04:08:39.062290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:39.061820    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt" event={"ID":"0d800e6c-4e04-4b08-a196-6522516d4214","Type":"ContainerDied","Data":"1f0745299458b6bdf0d8163e69f0779838df766c4c08b5def42fb1ef5339cbe6"}
Oct 31 04:08:39.063798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:39.063779    6931 generic.go:356] "Generic (PLEG): container finished" podID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerID="400be9661aa6c2af274f31d0889c6f8234b6e450e1497a1ee13105d48bac8d24" exitCode=0
Oct 31 04:08:39.063894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:39.063860    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r" event={"ID":"15d5b4fc-a21c-409e-bb47-1c81139b8639","Type":"ContainerDied","Data":"400be9661aa6c2af274f31d0889c6f8234b6e450e1497a1ee13105d48bac8d24"}
Oct 31 04:08:40.207497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.207473    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:40.210543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.210521    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:40.257344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257313    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\" (UniqueName: \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\") pod \"15d5b4fc-a21c-409e-bb47-1c81139b8639\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") "
Oct 31 04:08:40.257507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257393    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\" (UniqueName: \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\") pod \"0d800e6c-4e04-4b08-a196-6522516d4214\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") "
Oct 31 04:08:40.257507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257413    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\") pod \"15d5b4fc-a21c-409e-bb47-1c81139b8639\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") "
Oct 31 04:08:40.257507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257459    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\") pod \"15d5b4fc-a21c-409e-bb47-1c81139b8639\" (UID: \"15d5b4fc-a21c-409e-bb47-1c81139b8639\") "
Oct 31 04:08:40.257507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257479    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\") pod \"0d800e6c-4e04-4b08-a196-6522516d4214\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") "
Oct 31 04:08:40.257733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.257509    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\") pod \"0d800e6c-4e04-4b08-a196-6522516d4214\" (UID: \"0d800e6c-4e04-4b08-a196-6522516d4214\") "
Oct 31 04:08:40.258145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.258117    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle" pod "15d5b4fc-a21c-409e-bb47-1c81139b8639" (UID: "15d5b4fc-a21c-409e-bb47-1c81139b8639"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:40.258370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.258348    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle" pod "0d800e6c-4e04-4b08-a196-6522516d4214" (UID: "0d800e6c-4e04-4b08-a196-6522516d4214"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:40.260138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.260108    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4" pod "0d800e6c-4e04-4b08-a196-6522516d4214" (UID: "0d800e6c-4e04-4b08-a196-6522516d4214"). InnerVolumeSpecName "kube-api-access-xrkz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:40.260305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.260143    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q" pod "15d5b4fc-a21c-409e-bb47-1c81139b8639" (UID: "15d5b4fc-a21c-409e-bb47-1c81139b8639"). InnerVolumeSpecName "kube-api-access-pp58q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:40.261973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.261945    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util" pod "0d800e6c-4e04-4b08-a196-6522516d4214" (UID: "0d800e6c-4e04-4b08-a196-6522516d4214"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:40.262850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.262829    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util" pod "15d5b4fc-a21c-409e-bb47-1c81139b8639" (UID: "15d5b4fc-a21c-409e-bb47-1c81139b8639"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358625    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrkz4\" (UniqueName: \"kubernetes.io/projected/0d800e6c-4e04-4b08-a196-6522516d4214-kube-api-access-xrkz4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358669    6931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-util\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358683    6931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15d5b4fc-a21c-409e-bb47-1c81139b8639-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358694    6931 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-util\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358705    6931 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d800e6c-4e04-4b08-a196-6522516d4214-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.358729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:40.358715    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp58q\" (UniqueName: \"kubernetes.io/projected/15d5b4fc-a21c-409e-bb47-1c81139b8639-kube-api-access-pp58q\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:08:40.757521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:08:40.757432    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:08:41.073276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.073245    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt" event={"ID":"0d800e6c-4e04-4b08-a196-6522516d4214","Type":"ContainerDied","Data":"ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2"}
Oct 31 04:08:41.073276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.073272    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbgttt"
Oct 31 04:08:41.073520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.073294    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebfddb3f552c1e7161a2a1af590ee38d9f82a55db8d80ab1080b4ad942ebeaf2"
Oct 31 04:08:41.075141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.075111    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r" event={"ID":"15d5b4fc-a21c-409e-bb47-1c81139b8639","Type":"ContainerDied","Data":"0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f"}
Oct 31 04:08:41.075265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.075152    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mh94r"
Oct 31 04:08:41.075265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:41.075164    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c8d5d1001b411a2a480b0e8f231e7104011b35320ff692f5b93ddc722ae301f"
Oct 31 04:08:44.504077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:44.504040    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:46.757250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.757210    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:46.977400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977361    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"]
Oct 31 04:08:46.977752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977735    6931 cpu_manager.go:414] "containerMap: removing container" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="util"
Oct 31 04:08:46.977829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977754    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="util"
Oct 31 04:08:46.977829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977774    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="util"
Oct 31 04:08:46.977829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977782    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="util"
Oct 31 04:08:46.977829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977795    6931 cpu_manager.go:414] "containerMap: removing container" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="pull"
Oct 31 04:08:46.977829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977803    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="pull"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977849    6931 cpu_manager.go:414] "containerMap: removing container" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="extract"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977857    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="extract"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977867    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="extract"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977874    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="extract"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977891    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="pull"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977898    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="pull"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977971    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d5b4fc-a21c-409e-bb47-1c81139b8639" containerName="extract"
Oct 31 04:08:46.978080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.977980    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d800e6c-4e04-4b08-a196-6522516d4214" containerName="extract"
Oct 31 04:08:46.980023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.980004    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:46.992830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:46.992808    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"]
Oct 31 04:08:47.006612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.006587    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 04:08:47.006612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.006587    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:08:47.006612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.006609    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xsrv7\""
Oct 31 04:08:47.117144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.117103    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e080046-f2c3-4de1-a352-7c5dd6f70f08-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.117361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.117222    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67qt\" (UniqueName: \"kubernetes.io/projected/8e080046-f2c3-4de1-a352-7c5dd6f70f08-kube-api-access-c67qt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.218329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.218294    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e080046-f2c3-4de1-a352-7c5dd6f70f08-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.218508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.218349    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67qt\" (UniqueName: \"kubernetes.io/projected/8e080046-f2c3-4de1-a352-7c5dd6f70f08-kube-api-access-c67qt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.218765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.218745    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e080046-f2c3-4de1-a352-7c5dd6f70f08-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.243693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.243669    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67qt\" (UniqueName: \"kubernetes.io/projected/8e080046-f2c3-4de1-a352-7c5dd6f70f08-kube-api-access-c67qt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rspmg\" (UID: \"8e080046-f2c3-4de1-a352-7c5dd6f70f08\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.290651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.290611    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"
Oct 31 04:08:47.624489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:47.624458    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg"]
Oct 31 04:08:47.643870 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:47.643843    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e080046_f2c3_4de1_a352_7c5dd6f70f08.slice/crio-c2bed2bb158422733ab5a8858d05d11e24a23f2a58e8c37426fc1a745a939949 WatchSource:0}: Error finding container c2bed2bb158422733ab5a8858d05d11e24a23f2a58e8c37426fc1a745a939949: Status 404 returned error can't find the container with id c2bed2bb158422733ab5a8858d05d11e24a23f2a58e8c37426fc1a745a939949
Oct 31 04:08:48.103519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:48.103482    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg" event={"ID":"8e080046-f2c3-4de1-a352-7c5dd6f70f08","Type":"ContainerStarted","Data":"c2bed2bb158422733ab5a8858d05d11e24a23f2a58e8c37426fc1a745a939949"}
Oct 31 04:08:49.244862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:49.244801    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:49.450214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:49.450148    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:49.801005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:49.800973    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:50.051815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:50.051732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:50.113813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:50.113777    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg" event={"ID":"8e080046-f2c3-4de1-a352-7c5dd6f70f08","Type":"ContainerStarted","Data":"637da23793b35d5e93695e801a1e2a8eddd91902468a9eca51015b267776584a"}
Oct 31 04:08:50.187547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:50.187493    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rspmg" podStartSLOduration=2.506693204 podStartE2EDuration="4.187479476s" totalImagesPullingTime="1.680786272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:50.187278356 +0000 UTC m=+226.220278054" watchObservedRunningTime="2026-10-31 04:08:50.187479476 +0000 UTC m=+226.220479153"
Oct 31 04:08:51.253852 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.253811    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:51.314327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.314292    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"]
Oct 31 04:08:51.316886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.316862    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.351105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.351064    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"]
Oct 31 04:08:51.411713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.411662    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 04:08:51.411890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.411772    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 04:08:51.411890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.411804    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 04:08:51.411987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.411969    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 04:08:51.412054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.412042    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 04:08:51.412126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.412116    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cbq7v\""
Oct 31 04:08:51.459925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.459885    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.460109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.459985    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.460109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.460024    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.460245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.460137    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk68g\" (UniqueName: \"kubernetes.io/projected/4aeeedf2-1aae-4569-89be-d897171101c5-kube-api-access-mk68g\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.460245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.460217    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.460318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.460243    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.493962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.493928    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:51.561125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561089    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561130    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561179    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk68g\" (UniqueName: \"kubernetes.io/projected/4aeeedf2-1aae-4569-89be-d897171101c5-kube-api-access-mk68g\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561201    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561222    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561507    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.561641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.561617    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-manifests\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.563570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.563544    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeeedf2-1aae-4569-89be-d897171101c5-tmp\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.563824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.563801    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.563915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.563892    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.563972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.563904    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4aeeedf2-1aae-4569-89be-d897171101c5-metrics-cert\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.592467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.592432    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk68g\" (UniqueName: \"kubernetes.io/projected/4aeeedf2-1aae-4569-89be-d897171101c5-kube-api-access-mk68g\") pod \"opendatahub-operator-controller-manager-79b6bb9d79-r29jd\" (UID: \"4aeeedf2-1aae-4569-89be-d897171101c5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.630418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.630381    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:51.779327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.779300    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:51.995926 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:51.995833    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"]
Oct 31 04:08:51.999908 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:51.999878    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeeedf2_1aae_4569_89be_d897171101c5.slice/crio-3e2a366bb7fddd43cffdaefc85d9a49446b0de253001d13fdb2a832799dc5de9 WatchSource:0}: Error finding container 3e2a366bb7fddd43cffdaefc85d9a49446b0de253001d13fdb2a832799dc5de9: Status 404 returned error can't find the container with id 3e2a366bb7fddd43cffdaefc85d9a49446b0de253001d13fdb2a832799dc5de9
Oct 31 04:08:52.014364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.014333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:52.123411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.123374    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd" event={"ID":"4aeeedf2-1aae-4569-89be-d897171101c5","Type":"ContainerStarted","Data":"3e2a366bb7fddd43cffdaefc85d9a49446b0de253001d13fdb2a832799dc5de9"}
Oct 31 04:08:52.327124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.327079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:52.548047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.547998    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:52.573877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.573848    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"]
Oct 31 04:08:52.577490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.577433    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.589351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.589325    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"]
Oct 31 04:08:52.592580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.592551    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 04:08:52.592701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.592610    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 04:08:52.616450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.616426    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m66vt\""
Oct 31 04:08:52.673968 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.673933    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.674188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.673994    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6kq\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-kube-api-access-6p6kq\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.775083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.775045    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6kq\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-kube-api-access-6p6kq\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.775280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.775210    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.799021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.798986    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.812194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.812149    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6kq\" (UniqueName: \"kubernetes.io/projected/9832c9e3-d11d-4b59-b20c-85a036b297b2-kube-api-access-6p6kq\") pod \"cert-manager-webhook-5dbbbf75cc-6wsnk\" (UID: \"9832c9e3-d11d-4b59-b20c-85a036b297b2\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:52.904977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:52.904882    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:53.220886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:53.220799    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"]
Oct 31 04:08:53.226871 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:53.226840    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9832c9e3_d11d_4b59_b20c_85a036b297b2.slice/crio-34e993d749ba6c1ee84dfc35f307d73ea60c7413014f3e1f77f926770a821682 WatchSource:0}: Error finding container 34e993d749ba6c1ee84dfc35f307d73ea60c7413014f3e1f77f926770a821682: Status 404 returned error can't find the container with id 34e993d749ba6c1ee84dfc35f307d73ea60c7413014f3e1f77f926770a821682
Oct 31 04:08:53.849474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:53.849434    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:54.060440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.060401    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:54.134914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.134837    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk" event={"ID":"9832c9e3-d11d-4b59-b20c-85a036b297b2","Type":"ContainerStarted","Data":"34e993d749ba6c1ee84dfc35f307d73ea60c7413014f3e1f77f926770a821682"}
Oct 31 04:08:54.204668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.203211    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"]
Oct 31 04:08:54.206779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.206752    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.228289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.228261    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"]
Oct 31 04:08:54.246970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.246941    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qvztk\""
Oct 31 04:08:54.292249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.292201    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrpw\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-kube-api-access-2hrpw\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.292431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.292327    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.393923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.393842    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrpw\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-kube-api-access-2hrpw\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.393923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.393893    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.421220 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.421188    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrpw\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-kube-api-access-2hrpw\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.429191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.429142    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d15c1fc2-647a-4bb3-9ba3-c047e086cc17-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rcj64\" (UID: \"d15c1fc2-647a-4bb3-9ba3-c047e086cc17\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.521070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.521031    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"
Oct 31 04:08:54.621400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.621366    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:54.828867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.828821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:54.831471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:54.831443    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64"]
Oct 31 04:08:54.839202 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:08:54.836494    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd15c1fc2_647a_4bb3_9ba3_c047e086cc17.slice/crio-9ab9b075a2e6b5adee3a1eac5b922c208c9febd3634f65067debe1b7277242f6 WatchSource:0}: Error finding container 9ab9b075a2e6b5adee3a1eac5b922c208c9febd3634f65067debe1b7277242f6: Status 404 returned error can't find the container with id 9ab9b075a2e6b5adee3a1eac5b922c208c9febd3634f65067debe1b7277242f6
Oct 31 04:08:55.057825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:55.057793    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:55.140993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:55.140955    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd" event={"ID":"4aeeedf2-1aae-4569-89be-d897171101c5","Type":"ContainerStarted","Data":"d4942d47e32a2fecf18ee2cf65b010cca54acdde6d3ad83464c77312a8cd6c75"}
Oct 31 04:08:55.142279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:55.142254    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64" event={"ID":"d15c1fc2-647a-4bb3-9ba3-c047e086cc17","Type":"ContainerStarted","Data":"9ab9b075a2e6b5adee3a1eac5b922c208c9febd3634f65067debe1b7277242f6"}
Oct 31 04:08:56.150066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:56.150019    6931 generic.go:356] "Generic (PLEG): container finished" podID="4aeeedf2-1aae-4569-89be-d897171101c5" containerID="d4942d47e32a2fecf18ee2cf65b010cca54acdde6d3ad83464c77312a8cd6c75" exitCode=0
Oct 31 04:08:56.150577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:56.150119    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd" event={"ID":"4aeeedf2-1aae-4569-89be-d897171101c5","Type":"ContainerDied","Data":"d4942d47e32a2fecf18ee2cf65b010cca54acdde6d3ad83464c77312a8cd6c75"}
Oct 31 04:08:57.000299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.000266    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:57.161908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.161873    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk" event={"ID":"9832c9e3-d11d-4b59-b20c-85a036b297b2","Type":"ContainerStarted","Data":"00d0c9a73ef7a6a7971970ee405e6d6a512da95c585e5355fb1b208aec5cff1b"}
Oct 31 04:08:57.162375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.162082    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:08:57.163503 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.163483    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64" event={"ID":"d15c1fc2-647a-4bb3-9ba3-c047e086cc17","Type":"ContainerStarted","Data":"2a89df349cb549bd95c53820f03e62823ea75f32cf85cc980f8a5b3f3aea5c5c"}
Oct 31 04:08:57.165266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.165244    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd" event={"ID":"4aeeedf2-1aae-4569-89be-d897171101c5","Type":"ContainerStarted","Data":"59417d2c41b4aaecd744b169afbab4d32e56ef7dd0bccaea107b3664cc537b6d"}
Oct 31 04:08:57.165383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.165369    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:08:57.201771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.201722    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk" podStartSLOduration=2.161711762 podStartE2EDuration="5.201711008s" totalImagesPullingTime="3.039999246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:57.201447002 +0000 UTC m=+233.234446706" watchObservedRunningTime="2026-10-31 04:08:57.201711008 +0000 UTC m=+233.234710687"
Oct 31 04:08:57.243394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.243360    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:57.300502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.300453    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd" podStartSLOduration=2.338482808 podStartE2EDuration="6.300440052s" totalImagesPullingTime="3.349150885s" totalInitContainerRuntime="612.806359ms" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:57.300001254 +0000 UTC m=+233.333000998" watchObservedRunningTime="2026-10-31 04:08:57.300440052 +0000 UTC m=+233.333439728"
Oct 31 04:08:57.342399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:57.342352    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rcj64" podStartSLOduration=1.626218492 podStartE2EDuration="3.342338463s" totalImagesPullingTime="1.716119971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:57.341946878 +0000 UTC m=+233.374946577" watchObservedRunningTime="2026-10-31 04:08:57.342338463 +0000 UTC m=+233.375338142"
Oct 31 04:08:59.781145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:08:59.781111    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:01.393832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:01.393797    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:03.157394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.157355    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b5jf6"]
Oct 31 04:09:03.217717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.217678    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b5jf6"]
Oct 31 04:09:03.217881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.217747    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6wsnk"
Oct 31 04:09:03.217881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.217854    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.248330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.248303    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d6stv\""
Oct 31 04:09:03.378106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.378064    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7xs\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-kube-api-access-tr7xs\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.378334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.378126    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-bound-sa-token\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.479734 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.479638    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7xs\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-kube-api-access-tr7xs\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.479734 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.479698    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-bound-sa-token\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.504436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.504407    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-bound-sa-token\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.504588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.504491    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7xs\" (UniqueName: \"kubernetes.io/projected/baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b-kube-api-access-tr7xs\") pod \"cert-manager-597b95cb84-b5jf6\" (UID: \"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b\") " pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.545966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.545935    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b5jf6"
Oct 31 04:09:03.797661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.797633    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:03.863898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:03.863866    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b5jf6"]
Oct 31 04:09:03.868958 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:09:03.868935    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf64f0e_3fd0_4fd8_af6f_78d76bfc8a8b.slice/crio-33fcb7a99db8187b9792d607ab2ae138d1bb040fb73883af6a12aacbb59dffe4 WatchSource:0}: Error finding container 33fcb7a99db8187b9792d607ab2ae138d1bb040fb73883af6a12aacbb59dffe4: Status 404 returned error can't find the container with id 33fcb7a99db8187b9792d607ab2ae138d1bb040fb73883af6a12aacbb59dffe4
Oct 31 04:09:04.207511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:04.207416    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b5jf6" event={"ID":"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b","Type":"ContainerStarted","Data":"14e486bb21bdffd76bd9e0709e7166540d675f5c948b1c5133cafba358abf406"}
Oct 31 04:09:04.207511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:04.207467    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b5jf6" event={"ID":"baf64f0e-3fd0-4fd8-af6f-78d76bfc8a8b","Type":"ContainerStarted","Data":"33fcb7a99db8187b9792d607ab2ae138d1bb040fb73883af6a12aacbb59dffe4"}
Oct 31 04:09:04.246664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:04.246595    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b5jf6" podStartSLOduration=1.246579531 podStartE2EDuration="1.246579531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:04.245763364 +0000 UTC m=+240.278763044" watchObservedRunningTime="2026-10-31 04:09:04.246579531 +0000 UTC m=+240.279579210"
Oct 31 04:09:04.623072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:04.623039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:05.233524 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:05.233492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:05.459730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:05.459701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:08.016727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.016687    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"]
Oct 31 04:09:08.020217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.020194    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.053703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.053666    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fnxsq\""
Oct 31 04:09:08.053873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.053710    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 04:09:08.053873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.053789    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 04:09:08.054900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.054876    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"]
Oct 31 04:09:08.116206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.116168    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8sx\" (UniqueName: \"kubernetes.io/projected/e07064a6-61e7-4205-8059-c19474b06653-kube-api-access-jp8sx\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.116394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.116272    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07064a6-61e7-4205-8059-c19474b06653-tmp\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.174138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.174109    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79b6bb9d79-r29jd"
Oct 31 04:09:08.216748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.216700    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8sx\" (UniqueName: \"kubernetes.io/projected/e07064a6-61e7-4205-8059-c19474b06653-kube-api-access-jp8sx\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.216748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.216753    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07064a6-61e7-4205-8059-c19474b06653-tmp\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.217260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.217239    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07064a6-61e7-4205-8059-c19474b06653-tmp\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.240018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.239976    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8sx\" (UniqueName: \"kubernetes.io/projected/e07064a6-61e7-4205-8059-c19474b06653-kube-api-access-jp8sx\") pod \"jobset-operator-674cd59dc5-qzt2p\" (UID: \"e07064a6-61e7-4205-8059-c19474b06653\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.331283 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.331245    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"
Oct 31 04:09:08.627027 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:08.626941    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p"]
Oct 31 04:09:08.632060 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:09:08.632034    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode07064a6_61e7_4205_8059_c19474b06653.slice/crio-7fd71c4bf7f79482bb2bbd188e794850a4484a7d32854070d319976de3e3533d WatchSource:0}: Error finding container 7fd71c4bf7f79482bb2bbd188e794850a4484a7d32854070d319976de3e3533d: Status 404 returned error can't find the container with id 7fd71c4bf7f79482bb2bbd188e794850a4484a7d32854070d319976de3e3533d
Oct 31 04:09:09.236343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:09.236306    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p" event={"ID":"e07064a6-61e7-4205-8059-c19474b06653","Type":"ContainerStarted","Data":"7fd71c4bf7f79482bb2bbd188e794850a4484a7d32854070d319976de3e3533d"}
Oct 31 04:09:10.761424 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:09:10.761380    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:09:11.227566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:11.227533    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:12.252404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:12.252371    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p" event={"ID":"e07064a6-61e7-4205-8059-c19474b06653","Type":"ContainerStarted","Data":"863000e6a78abd5ec9da117a5918bf06ae3424b991b0190cbd732fef20d52d62"}
Oct 31 04:09:12.296255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:12.296151    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qzt2p" podStartSLOduration=1.8747083930000001 podStartE2EDuration="5.296134621s" totalImagesPullingTime="3.421426228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:12.295207528 +0000 UTC m=+248.328207229" watchObservedRunningTime="2026-10-31 04:09:12.296134621 +0000 UTC m=+248.329134301"
Oct 31 04:09:12.404879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:12.404789    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:13.076599 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:13.076504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:15.263851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:15.263814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:15.795168 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:15.795128    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:16.036512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:16.036475    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:16.297829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:16.297783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:18.030849 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:18.030799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:18.555447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:18.555412    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:18.790718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:18.790689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.027121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.027032    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.358886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.358850    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.544173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.544123    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-892tn"]
Oct 31 04:09:19.546986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.546965    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.562747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.562724    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 04:09:19.562870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.562765    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 04:09:19.576673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.576640    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-892tn"]
Oct 31 04:09:19.626061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.625964    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f8bf9a-4406-4cc3-986f-35c546130809-tls-certs\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.626061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.626027    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kv59\" (UniqueName: \"kubernetes.io/projected/25f8bf9a-4406-4cc3-986f-35c546130809-kube-api-access-8kv59\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.626282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.626119    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25f8bf9a-4406-4cc3-986f-35c546130809-tmp\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.634285 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.634261    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-flbxz\""
Oct 31 04:09:19.642745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.642725    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 04:09:19.642831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.642797    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 04:09:19.672203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.672145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.726892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.726854    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f8bf9a-4406-4cc3-986f-35c546130809-tls-certs\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.727075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.726912    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kv59\" (UniqueName: \"kubernetes.io/projected/25f8bf9a-4406-4cc3-986f-35c546130809-kube-api-access-8kv59\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.727075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.726951    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25f8bf9a-4406-4cc3-986f-35c546130809-tmp\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.729074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.729049    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25f8bf9a-4406-4cc3-986f-35c546130809-tmp\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.729478 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.729459    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f8bf9a-4406-4cc3-986f-35c546130809-tls-certs\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.750606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.750574    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kv59\" (UniqueName: \"kubernetes.io/projected/25f8bf9a-4406-4cc3-986f-35c546130809-kube-api-access-8kv59\") pod \"kube-auth-proxy-58666cb9df-892tn\" (UID: \"25f8bf9a-4406-4cc3-986f-35c546130809\") " pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.858022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.857990    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn"
Oct 31 04:09:19.867303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.867279    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"]
Oct 31 04:09:19.869725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.869706    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:19.889514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.889437    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 04:09:19.889514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.889437    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 04:09:19.889514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.889484    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 04:09:19.890859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.890835    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"]
Oct 31 04:09:19.899723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.899686    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.928261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.928232    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:19.928404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.928283    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shxj\" (UniqueName: \"kubernetes.io/projected/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-kube-api-access-5shxj\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:19.928404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.928305    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:19.944171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.944125    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 04:09:19.944305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:19.944173    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-86txz\""
Oct 31 04:09:20.029439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.029407    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.029624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.029483    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shxj\" (UniqueName: \"kubernetes.io/projected/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-kube-api-access-5shxj\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.029624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.029517    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.029624 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:09:20.029582    6931 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Oct 31 04:09:20.029747 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:09:20.029683    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert podName:faeda3ac-ef2c-4bd5-a821-465f8bae2cd8 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:20.529659547 +0000 UTC m=+256.562659227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-sqjb4" (UID: "faeda3ac-ef2c-4bd5-a821-465f8bae2cd8") : secret "plugin-serving-cert" not found
Oct 31 04:09:20.030234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.030215    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.052775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.052745    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shxj\" (UniqueName: \"kubernetes.io/projected/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-kube-api-access-5shxj\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.118574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.118540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:20.161499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.159614    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58666cb9df-892tn"]
Oct 31 04:09:20.166430 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:09:20.166404    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f8bf9a_4406_4cc3_986f_35c546130809.slice/crio-7e35ae1580ceeb7023ec731b1b9bced81c19cf4fd81ab2f9847675fa881ff773 WatchSource:0}: Error finding container 7e35ae1580ceeb7023ec731b1b9bced81c19cf4fd81ab2f9847675fa881ff773: Status 404 returned error can't find the container with id 7e35ae1580ceeb7023ec731b1b9bced81c19cf4fd81ab2f9847675fa881ff773
Oct 31 04:09:20.291237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.291197    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn" event={"ID":"25f8bf9a-4406-4cc3-986f-35c546130809","Type":"ContainerStarted","Data":"7e35ae1580ceeb7023ec731b1b9bced81c19cf4fd81ab2f9847675fa881ff773"}
Oct 31 04:09:20.335702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.335664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:20.535062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.535032    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.537433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.537413    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faeda3ac-ef2c-4bd5-a821-465f8bae2cd8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sqjb4\" (UID: \"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.548312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.548281    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:20.789023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.788941    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"
Oct 31 04:09:20.972496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.972464    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:09:20.975744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.975722    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:20.997964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:20.997937    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 04:09:21.003466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.003444    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:09:21.026091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.026044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:21.041105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.041032    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.041105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.041082    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56n9\" (UniqueName: \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.101904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.101866    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:09:21.108446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.108415    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4"]
Oct 31 04:09:21.114924 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:09:21.114894    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaeda3ac_ef2c_4bd5_a821_465f8bae2cd8.slice/crio-51c33694b2218873f2da5e61f926606ea73a91e58986ae4d66767cfcc4a199e7 WatchSource:0}: Error finding container 51c33694b2218873f2da5e61f926606ea73a91e58986ae4d66767cfcc4a199e7: Status 404 returned error can't find the container with id 51c33694b2218873f2da5e61f926606ea73a91e58986ae4d66767cfcc4a199e7
Oct 31 04:09:21.141619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.141584    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.141778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.141644    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56n9\" (UniqueName: \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.142365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.142342    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.165239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.165203    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56n9\" (UniqueName: \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\") pod \"limitador-limitador-74dfcfb894-4l98m\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.244594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.244552    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:21.291423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.291338    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:21.298368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.298338    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4" event={"ID":"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8","Type":"ContainerStarted","Data":"51c33694b2218873f2da5e61f926606ea73a91e58986ae4d66767cfcc4a199e7"}
Oct 31 04:09:21.464709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.464674    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:21.594272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:21.594187    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:09:21.600455 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:09:21.600419    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139bfca9_b033_4423_9f2a_a7f812086282.slice/crio-83d6a8c84514241e290731d550873957359c63efb84c8c9bc95c33a9d56df39e WatchSource:0}: Error finding container 83d6a8c84514241e290731d550873957359c63efb84c8c9bc95c33a9d56df39e: Status 404 returned error can't find the container with id 83d6a8c84514241e290731d550873957359c63efb84c8c9bc95c33a9d56df39e
Oct 31 04:09:22.266257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:22.266206    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:22.309781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:22.309744    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" event={"ID":"139bfca9-b033-4423-9f2a-a7f812086282","Type":"ContainerStarted","Data":"83d6a8c84514241e290731d550873957359c63efb84c8c9bc95c33a9d56df39e"}
Oct 31 04:09:23.317137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:23.317094    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn" event={"ID":"25f8bf9a-4406-4cc3-986f-35c546130809","Type":"ContainerStarted","Data":"f293143149486b4c052197a9e34b0db0af0c37ad1b7a2cadb63afb2b1e4cdfdd"}
Oct 31 04:09:23.362202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:23.362123    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58666cb9df-892tn" podStartSLOduration=1.504789455 podStartE2EDuration="4.362108575s" totalImagesPullingTime="2.85731912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:23.361542933 +0000 UTC m=+259.394542614" watchObservedRunningTime="2026-10-31 04:09:23.362108575 +0000 UTC m=+259.395108253"
Oct 31 04:09:24.691548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:24.691460    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:24.901335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:24.901287    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.329568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:25.329523    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" event={"ID":"139bfca9-b033-4423-9f2a-a7f812086282","Type":"ContainerStarted","Data":"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"}
Oct 31 04:09:25.329752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:25.329642    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:25.371133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:25.371079    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" podStartSLOduration=2.584350399 podStartE2EDuration="5.371066045s" totalImagesPullingTime="2.786715646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:25.370401278 +0000 UTC m=+261.403400979" watchObservedRunningTime="2026-10-31 04:09:25.371066045 +0000 UTC m=+261.404065716"
Oct 31 04:09:25.603530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:25.603439    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.805989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:25.805947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:26.019909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:26.019826    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:26.335535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:26.335494    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4" event={"ID":"faeda3ac-ef2c-4bd5-a821-465f8bae2cd8","Type":"ContainerStarted","Data":"bd637eb8cc308e4c6ab7beceb6c99264eb33a652b271508110efcb1c5d454606"}
Oct 31 04:09:26.375256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:26.375206    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sqjb4" podStartSLOduration=2.293513136 podStartE2EDuration="7.375192436s" totalImagesPullingTime="5.0816793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:26.374461675 +0000 UTC m=+262.407461374" watchObservedRunningTime="2026-10-31 04:09:26.375192436 +0000 UTC m=+262.408192114"
Oct 31 04:09:26.393693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:26.393664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:26.628003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:26.627912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:34.598816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:34.598782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:34.805603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:34.805567    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:36.337051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:36.337024    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:09:40.014776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:40.014739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:40.235869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:40.235821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:40.759690 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:09:40.759648    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:09:41.890597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:41.890561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:42.110574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:42.110535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:42.651113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:42.651083    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:42.876773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:42.876670    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:46.226066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:46.225969    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:46.434010 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:46.433976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:46.643560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:46.643524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:46.848512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:46.848480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:48.449064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:48.449025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:48.656240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:48.656200    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:52.881740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:52.881707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:53.103771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:53.103734    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:55.013173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:55.013127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:59.427004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:59.426969    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:59.647073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:09:59.647029    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:01.266583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.266540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:01.473913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.473871    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:01.921812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.921774    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:01.945822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.945790    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:01.945975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.945902    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:01.960557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:01.960530    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hc5c9\""
Oct 31 04:10:02.041201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.041140    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqz7\" (UniqueName: \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\") pod \"authorino-57f7f8659f-z7gvh\" (UID: \"af51d3c1-ff84-46e7-afeb-f8cf18676e23\") " pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:02.093394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.093363    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:02.142403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.142367    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqz7\" (UniqueName: \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\") pod \"authorino-57f7f8659f-z7gvh\" (UID: \"af51d3c1-ff84-46e7-afeb-f8cf18676e23\") " pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:02.164991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.164957    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqz7\" (UniqueName: \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\") pod \"authorino-57f7f8659f-z7gvh\" (UID: \"af51d3c1-ff84-46e7-afeb-f8cf18676e23\") " pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:02.258183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.258078    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:02.307491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.307369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:02.549656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.549618    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:10:02.553541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.553521    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:10:02.553653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.553598    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:02.558934 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:02.558910    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf51d3c1_ff84_46e7_afeb_f8cf18676e23.slice/crio-8d3670679f30f2051427786d69585491690753bbc28738cb819128b56a924800 WatchSource:0}: Error finding container 8d3670679f30f2051427786d69585491690753bbc28738cb819128b56a924800: Status 404 returned error can't find the container with id 8d3670679f30f2051427786d69585491690753bbc28738cb819128b56a924800
Oct 31 04:10:02.571044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.571019    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:10:02.646459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.646412    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\") pod \"authorino-56c6c9ddd5-mgrwq\" (UID: \"26b755b6-314e-4022-9c5c-3f55d68d3f3a\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:10:02.747909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.747865    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\") pod \"authorino-56c6c9ddd5-mgrwq\" (UID: \"26b755b6-314e-4022-9c5c-3f55d68d3f3a\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:10:02.770675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.770646    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\") pod \"authorino-56c6c9ddd5-mgrwq\" (UID: \"26b755b6-314e-4022-9c5c-3f55d68d3f3a\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:10:02.875496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:02.875412    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:10:03.170113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:03.170030    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:10:03.174842 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:03.174816    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26b755b6_314e_4022_9c5c_3f55d68d3f3a.slice/crio-5081c8ef7f765656218f8c3b9ab1a0fe1649c8993e614e0908ced6add5cb9e6d WatchSource:0}: Error finding container 5081c8ef7f765656218f8c3b9ab1a0fe1649c8993e614e0908ced6add5cb9e6d: Status 404 returned error can't find the container with id 5081c8ef7f765656218f8c3b9ab1a0fe1649c8993e614e0908ced6add5cb9e6d
Oct 31 04:10:03.506043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:03.505956    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" event={"ID":"26b755b6-314e-4022-9c5c-3f55d68d3f3a","Type":"ContainerStarted","Data":"5081c8ef7f765656218f8c3b9ab1a0fe1649c8993e614e0908ced6add5cb9e6d"}
Oct 31 04:10:03.507460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:03.507432    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" event={"ID":"af51d3c1-ff84-46e7-afeb-f8cf18676e23","Type":"ContainerStarted","Data":"8d3670679f30f2051427786d69585491690753bbc28738cb819128b56a924800"}
Oct 31 04:10:04.513593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:04.513549    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" event={"ID":"26b755b6-314e-4022-9c5c-3f55d68d3f3a","Type":"ContainerStarted","Data":"4fc7ceec1ae33974b80f899bc6a02dd26d3140e3c863f75b68b1004e89fbc52e"}
Oct 31 04:10:04.515111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:04.515086    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" event={"ID":"af51d3c1-ff84-46e7-afeb-f8cf18676e23","Type":"ContainerStarted","Data":"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"}
Oct 31 04:10:04.551175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:04.551089    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" podStartSLOduration=1.342121889 podStartE2EDuration="2.551070493s" totalImagesPullingTime="1.208948604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:04.551018129 +0000 UTC m=+300.584017809" watchObservedRunningTime="2026-10-31 04:10:04.551070493 +0000 UTC m=+300.584070169"
Oct 31 04:10:04.578046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:04.578012    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:04.590051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:04.589397    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" podStartSLOduration=1.764341098 podStartE2EDuration="3.589375889s" totalImagesPullingTime="1.825034791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:04.587821214 +0000 UTC m=+300.620820914" watchObservedRunningTime="2026-10-31 04:10:04.589375889 +0000 UTC m=+300.622375570"
Oct 31 04:10:06.524766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:06.524708    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" podUID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" containerName="authorino" containerID="cri-o://6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989" gracePeriod=30
Oct 31 04:10:06.762248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:06.762225    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:06.885544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:06.885514    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\" (UniqueName: \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\") pod \"af51d3c1-ff84-46e7-afeb-f8cf18676e23\" (UID: \"af51d3c1-ff84-46e7-afeb-f8cf18676e23\") "
Oct 31 04:10:06.887709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:06.887677    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7" pod "af51d3c1-ff84-46e7-afeb-f8cf18676e23" (UID: "af51d3c1-ff84-46e7-afeb-f8cf18676e23"). InnerVolumeSpecName "kube-api-access-2nqz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:06.986409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:06.986372    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqz7\" (UniqueName: \"kubernetes.io/projected/af51d3c1-ff84-46e7-afeb-f8cf18676e23-kube-api-access-2nqz7\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:10:07.529877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.529840    6931 generic.go:356] "Generic (PLEG): container finished" podID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" containerID="6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989" exitCode=0
Oct 31 04:10:07.530409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.529887    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7gvh"
Oct 31 04:10:07.530409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.529924    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" event={"ID":"af51d3c1-ff84-46e7-afeb-f8cf18676e23","Type":"ContainerDied","Data":"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"}
Oct 31 04:10:07.530409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.529981    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7gvh" event={"ID":"af51d3c1-ff84-46e7-afeb-f8cf18676e23","Type":"ContainerDied","Data":"8d3670679f30f2051427786d69585491690753bbc28738cb819128b56a924800"}
Oct 31 04:10:07.530409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.529999    6931 scope.go:122] "RemoveContainer" containerID="6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"
Oct 31 04:10:07.540702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.538941    6931 scope.go:122] "RemoveContainer" containerID="6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"
Oct 31 04:10:07.540813 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:10:07.540766    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989\": container with ID starting with 6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989 not found: ID does not exist" containerID="6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"
Oct 31 04:10:07.540870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.540806    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989"} err="failed to get container status \"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989\": rpc error: code = NotFound desc = could not find container \"6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989\": container with ID starting with 6e49e99ba3052f67c9f84a67963860a849692121cb0e63c16df15744926a2989 not found: ID does not exist"
Oct 31 04:10:07.579130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.579097    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:07.581143 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:07.581116    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7gvh"]
Oct 31 04:10:08.120622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:08.120590    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" path="/var/lib/kubelet/pods/af51d3c1-ff84-46e7-afeb-f8cf18676e23/volumes"
Oct 31 04:10:10.757819 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:10:10.757779    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:10:15.804874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:15.804837    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:10:15.805345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:15.805082    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" podUID="139bfca9-b033-4423-9f2a-a7f812086282" containerName="limitador" containerID="cri-o://6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8" gracePeriod=30
Oct 31 04:10:16.364187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.364118    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:10:16.473507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.473472    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\" (UniqueName: \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\") pod \"139bfca9-b033-4423-9f2a-a7f812086282\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") "
Oct 31 04:10:16.473694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.473517    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\" (UniqueName: \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\") pod \"139bfca9-b033-4423-9f2a-a7f812086282\" (UID: \"139bfca9-b033-4423-9f2a-a7f812086282\") "
Oct 31 04:10:16.473869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.473848    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file" pod "139bfca9-b033-4423-9f2a-a7f812086282" (UID: "139bfca9-b033-4423-9f2a-a7f812086282"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:16.475716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.475689    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9" pod "139bfca9-b033-4423-9f2a-a7f812086282" (UID: "139bfca9-b033-4423-9f2a-a7f812086282"). InnerVolumeSpecName "kube-api-access-q56n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:16.574293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.574263    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56n9\" (UniqueName: \"kubernetes.io/projected/139bfca9-b033-4423-9f2a-a7f812086282-kube-api-access-q56n9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:10:16.574293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.574289    6931 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/139bfca9-b033-4423-9f2a-a7f812086282-config-file\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:10:16.575205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.575177    6931 generic.go:356] "Generic (PLEG): container finished" podID="139bfca9-b033-4423-9f2a-a7f812086282" containerID="6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8" exitCode=0
Oct 31 04:10:16.575279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.575224    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m"
Oct 31 04:10:16.575279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.575267    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" event={"ID":"139bfca9-b033-4423-9f2a-a7f812086282","Type":"ContainerDied","Data":"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"}
Oct 31 04:10:16.575351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.575315    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" event={"ID":"139bfca9-b033-4423-9f2a-a7f812086282","Type":"ContainerDied","Data":"83d6a8c84514241e290731d550873957359c63efb84c8c9bc95c33a9d56df39e"}
Oct 31 04:10:16.575351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.575341    6931 scope.go:122] "RemoveContainer" containerID="6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"
Oct 31 04:10:16.584391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.584375    6931 scope.go:122] "RemoveContainer" containerID="6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"
Oct 31 04:10:16.584656 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:10:16.584630    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8\": container with ID starting with 6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8 not found: ID does not exist" containerID="6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"
Oct 31 04:10:16.584718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.584663    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8"} err="failed to get container status \"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8\": rpc error: code = NotFound desc = could not find container \"6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8\": container with ID starting with 6270d32c2a87468ba792e612696265629ccb02a76241a7ab1b6ba659bb815bc8 not found: ID does not exist"
Oct 31 04:10:16.624143 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.624014    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:10:16.628149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:16.628119    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4l98m"]
Oct 31 04:10:18.120214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:18.120182    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139bfca9-b033-4423-9f2a-a7f812086282" path="/var/lib/kubelet/pods/139bfca9-b033-4423-9f2a-a7f812086282/volumes"
Oct 31 04:10:21.336222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:21.336129    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-limitador-74dfcfb894-4l98m" podUID="139bfca9-b033-4423-9f2a-a7f812086282" containerName="limitador" probeResult="failure" output="Get \"http://10.134.0.30:8080/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:10:22.973948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:22.973910    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.268715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.268630    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qdpw6"]
Oct 31 04:10:23.269134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269104    6931 cpu_manager.go:414] "containerMap: removing container" podUID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" containerName="authorino"
Oct 31 04:10:23.269134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269128    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" containerName="authorino"
Oct 31 04:10:23.269338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269205    6931 cpu_manager.go:414] "containerMap: removing container" podUID="139bfca9-b033-4423-9f2a-a7f812086282" containerName="limitador"
Oct 31 04:10:23.269338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269214    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139bfca9-b033-4423-9f2a-a7f812086282" containerName="limitador"
Oct 31 04:10:23.269338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269306    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="139bfca9-b033-4423-9f2a-a7f812086282" containerName="limitador"
Oct 31 04:10:23.269338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.269320    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="af51d3c1-ff84-46e7-afeb-f8cf18676e23" containerName="authorino"
Oct 31 04:10:23.273447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.273428    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.307989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.307934    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 04:10:23.307989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.307960    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bshsh\""
Oct 31 04:10:23.308257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.307996    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 04:10:23.308257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.308085    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 04:10:23.309388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.309368    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qdpw6"]
Oct 31 04:10:23.310504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.310483    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:10:23.311895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.311872    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.314187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.314152    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.332612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.332587    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:10:23.333947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.333927    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8dsh2\""
Oct 31 04:10:23.334203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.334186    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 04:10:23.339847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.339819    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9mrkv"]
Oct 31 04:10:23.343380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.343360    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.348794 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.348768    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0517decd-0685-403b-b09f-0f7334b1b426-cert\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.348901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.348807    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176be731-5be3-437f-a60f-ec36fc3de64a-tls-certs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.349029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.349008    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.349065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.349046    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499tw\" (UniqueName: \"kubernetes.io/projected/0517decd-0685-403b-b09f-0f7334b1b426-kube-api-access-499tw\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.349194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.349147    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7pn\" (UniqueName: \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.349256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.349209    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2zs\" (UniqueName: \"kubernetes.io/projected/176be731-5be3-437f-a60f-ec36fc3de64a-kube-api-access-hn2zs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.366835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.366810    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9mrkv"]
Oct 31 04:10:23.377555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.377534    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6tlz5\""
Oct 31 04:10:23.377555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.377534    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 04:10:23.393217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.393191    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5qwzx"]
Oct 31 04:10:23.396555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.396539    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.415749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.415723    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 04:10:23.416736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.416712    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 04:10:23.416878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.416855    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hbml4\""
Oct 31 04:10:23.425723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.425690    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5qwzx"]
Oct 31 04:10:23.449784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449755    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-cert\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.449923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449793    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhqqs\" (UniqueName: \"kubernetes.io/projected/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-kube-api-access-vhqqs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.449923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449883    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.449923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449919    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499tw\" (UniqueName: \"kubernetes.io/projected/0517decd-0685-403b-b09f-0f7334b1b426-kube-api-access-499tw\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.450029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449942    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-metrics-certs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.450029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449967    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7pn\" (UniqueName: \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.450029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.449984    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2zs\" (UniqueName: \"kubernetes.io/projected/176be731-5be3-437f-a60f-ec36fc3de64a-kube-api-access-hn2zs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.450116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.450075    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0517decd-0685-403b-b09f-0f7334b1b426-cert\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.450116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.450108    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176be731-5be3-437f-a60f-ec36fc3de64a-tls-certs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.452459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.452425    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.452556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.452456    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0517decd-0685-403b-b09f-0f7334b1b426-cert\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.452556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.452550    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176be731-5be3-437f-a60f-ec36fc3de64a-tls-certs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.475397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.475371    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2zs\" (UniqueName: \"kubernetes.io/projected/176be731-5be3-437f-a60f-ec36fc3de64a-kube-api-access-hn2zs\") pod \"model-serving-api-75bd89fdfc-9mrkv\" (UID: \"176be731-5be3-437f-a60f-ec36fc3de64a\") " pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.475512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.475375    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7pn\" (UniqueName: \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\") pod \"llmisvc-controller-manager-869d5dcc5d-wrscd\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.475512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.475456    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-499tw\" (UniqueName: \"kubernetes.io/projected/0517decd-0685-403b-b09f-0f7334b1b426-kube-api-access-499tw\") pod \"kserve-controller-manager-776cc66d47-qdpw6\" (UID: \"0517decd-0685-403b-b09f-0f7334b1b426\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.546573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.546527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.551299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.551279    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-cert\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.551406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.551315    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhqqs\" (UniqueName: \"kubernetes.io/projected/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-kube-api-access-vhqqs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.551406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.551357    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-metrics-certs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.553840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.553820    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-metrics-certs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.553923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.553877    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-cert\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.574026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.573998    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhqqs\" (UniqueName: \"kubernetes.io/projected/8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d-kube-api-access-vhqqs\") pod \"odh-model-controller-769875db56-5qwzx\" (UID: \"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d\") " pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.585883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.585861    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:23.626093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.626054    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:23.656663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.656615    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:23.709356 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.709324    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:23.839438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.839352    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.885759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.885729    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qdpw6"]
Oct 31 04:10:23.890756 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:23.890728    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0517decd_0685_403b_b09f_0f7334b1b426.slice/crio-8127b5ac8bfa5b35c7f73dc054e8b2da89db495e387a8e7929ba98b75606f50b WatchSource:0}: Error finding container 8127b5ac8bfa5b35c7f73dc054e8b2da89db495e387a8e7929ba98b75606f50b: Status 404 returned error can't find the container with id 8127b5ac8bfa5b35c7f73dc054e8b2da89db495e387a8e7929ba98b75606f50b
Oct 31 04:10:23.943269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.943238    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"]
Oct 31 04:10:23.948141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.948120    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:23.951142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.951119    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:10:23.957522 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:23.957502    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63dfb6a0_5f2e_4744_a8c7_a3d307fe2778.slice/crio-ebe30795b03b6a28bc71ecb70a2e87c9295b700832ab62c08c8918ac62892ddd WatchSource:0}: Error finding container ebe30795b03b6a28bc71ecb70a2e87c9295b700832ab62c08c8918ac62892ddd: Status 404 returned error can't find the container with id ebe30795b03b6a28bc71ecb70a2e87c9295b700832ab62c08c8918ac62892ddd
Oct 31 04:10:23.963861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.963845    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d5qt2\""
Oct 31 04:10:23.966429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.966406    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 04:10:23.966807 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.966785    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:10:23.967522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.967500    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:10:23.968340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.968315    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"]
Oct 31 04:10:23.969955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:23.969932    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9mrkv"]
Oct 31 04:10:24.024272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.024239    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5qwzx"]
Oct 31 04:10:24.029606 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:24.029577    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc7b10a_1f4d_4b8a_85b6_37bf95208d0d.slice/crio-e3fc7b7876b1d2f2c37a6f1fc164a29402646c4c4bf84fb7018fa36800629711 WatchSource:0}: Error finding container e3fc7b7876b1d2f2c37a6f1fc164a29402646c4c4bf84fb7018fa36800629711: Status 404 returned error can't find the container with id e3fc7b7876b1d2f2c37a6f1fc164a29402646c4c4bf84fb7018fa36800629711
Oct 31 04:10:24.056474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.056443    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2b8dc5b1-daac-4b97-a08f-c104a612e45a-data\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.056598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.056559    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnlc\" (UniqueName: \"kubernetes.io/projected/2b8dc5b1-daac-4b97-a08f-c104a612e45a-kube-api-access-hbnlc\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.135019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.134931    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:24.157042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.157008    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2b8dc5b1-daac-4b97-a08f-c104a612e45a-data\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.157223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.157099    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnlc\" (UniqueName: \"kubernetes.io/projected/2b8dc5b1-daac-4b97-a08f-c104a612e45a-kube-api-access-hbnlc\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.157550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.157530    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2b8dc5b1-daac-4b97-a08f-c104a612e45a-data\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.180076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.180043    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnlc\" (UniqueName: \"kubernetes.io/projected/2b8dc5b1-daac-4b97-a08f-c104a612e45a-kube-api-access-hbnlc\") pod \"postgres-7cdbf6dc44-fwgt5\" (UID: \"2b8dc5b1-daac-4b97-a08f-c104a612e45a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.263441 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.263409    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:24.567819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.567786    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"]
Oct 31 04:10:24.577629 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:10:24.577597    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8dc5b1_daac_4b97_a08f_c104a612e45a.slice/crio-5c8cc264962de5d0b45862261c6a92f7ffbdabcbd2dbaa4fa1acdd658ee35053 WatchSource:0}: Error finding container 5c8cc264962de5d0b45862261c6a92f7ffbdabcbd2dbaa4fa1acdd658ee35053: Status 404 returned error can't find the container with id 5c8cc264962de5d0b45862261c6a92f7ffbdabcbd2dbaa4fa1acdd658ee35053
Oct 31 04:10:24.619480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.619387    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5" event={"ID":"2b8dc5b1-daac-4b97-a08f-c104a612e45a","Type":"ContainerStarted","Data":"5c8cc264962de5d0b45862261c6a92f7ffbdabcbd2dbaa4fa1acdd658ee35053"}
Oct 31 04:10:24.622508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.622417    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv" event={"ID":"176be731-5be3-437f-a60f-ec36fc3de64a","Type":"ContainerStarted","Data":"8433e59aa3e60dbc8715d4ebdf0b07baa6dc0b66f7c498729dfc9deda22acb7c"}
Oct 31 04:10:24.624387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.624332    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" event={"ID":"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778","Type":"ContainerStarted","Data":"ebe30795b03b6a28bc71ecb70a2e87c9295b700832ab62c08c8918ac62892ddd"}
Oct 31 04:10:24.627336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.627278    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5qwzx" event={"ID":"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d","Type":"ContainerStarted","Data":"e3fc7b7876b1d2f2c37a6f1fc164a29402646c4c4bf84fb7018fa36800629711"}
Oct 31 04:10:24.628887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:24.628824    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6" event={"ID":"0517decd-0685-403b-b09f-0f7334b1b426","Type":"ContainerStarted","Data":"8127b5ac8bfa5b35c7f73dc054e8b2da89db495e387a8e7929ba98b75606f50b"}
Oct 31 04:10:25.636443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:25.636406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:25.862464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:25.862417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:26.078029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:26.077883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:27.659841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:27.659810    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv" event={"ID":"176be731-5be3-437f-a60f-ec36fc3de64a","Type":"ContainerStarted","Data":"2dfdcd65c68d8a11f3b35b1f495cbb61da0f9ce2f28ac519eeabedf81db06395"}
Oct 31 04:10:27.660239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:27.660086    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:27.703003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:27.702940    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv" podStartSLOduration=1.527241295 podStartE2EDuration="4.702920444s" totalImagesPullingTime="3.175679149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:27.700755765 +0000 UTC m=+323.733755466" watchObservedRunningTime="2026-10-31 04:10:27.702920444 +0000 UTC m=+323.735920126"
Oct 31 04:10:28.666501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.666454    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5qwzx" event={"ID":"8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d","Type":"ContainerStarted","Data":"e9d4af12a503ba68b216289429c1c86664abc7f6fdec85eee1175b0b000390a1"}
Oct 31 04:10:28.666953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.666605    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:28.668495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.668462    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6" event={"ID":"0517decd-0685-403b-b09f-0f7334b1b426","Type":"ContainerStarted","Data":"5580050d81dd94bde3cc59e011e02e51516de92c35653a93ee77e2d1fc3ff351"}
Oct 31 04:10:28.668623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.668524    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:10:28.670223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.670195    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" event={"ID":"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778","Type":"ContainerStarted","Data":"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"}
Oct 31 04:10:28.670451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.670403    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:28.706967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.706875    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5qwzx" podStartSLOduration=1.310008546 podStartE2EDuration="5.706857852s" totalImagesPullingTime="4.396849306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.706384851 +0000 UTC m=+324.739384551" watchObservedRunningTime="2026-10-31 04:10:28.706857852 +0000 UTC m=+324.739857546"
Oct 31 04:10:28.747251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.747186    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" podStartSLOduration=2.078159855 podStartE2EDuration="5.74716899s" totalImagesPullingTime="3.669009135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.746431234 +0000 UTC m=+324.779430928" watchObservedRunningTime="2026-10-31 04:10:28.74716899 +0000 UTC m=+324.780168664"
Oct 31 04:10:28.788366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:28.788310    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6" podStartSLOduration=1.699722704 podStartE2EDuration="5.788292905s" totalImagesPullingTime="4.088570201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.787288992 +0000 UTC m=+324.820288672" watchObservedRunningTime="2026-10-31 04:10:28.788292905 +0000 UTC m=+324.821292583"
Oct 31 04:10:31.687361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:31.687324    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5" event={"ID":"2b8dc5b1-daac-4b97-a08f-c104a612e45a","Type":"ContainerStarted","Data":"6097b2cf92f7e911f51b69e741af3076d5129b574e949e6702373356f5acb2dd"}
Oct 31 04:10:31.687739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:31.687477    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:31.727869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:31.727808    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5" podStartSLOduration=2.513830973 podStartE2EDuration="8.727793813s" totalImagesPullingTime="6.21396284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:31.726191697 +0000 UTC m=+327.759191371" watchObservedRunningTime="2026-10-31 04:10:31.727793813 +0000 UTC m=+327.760793492"
Oct 31 04:10:37.723262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:37.723230    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fwgt5"
Oct 31 04:10:38.674632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:38.674602    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-9mrkv"
Oct 31 04:10:39.678583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:39.678552    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5qwzx"
Oct 31 04:10:40.758891 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:10:40.758853    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:10:46.366900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:46.366866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:46.579286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:46.579248    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:46.790072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:46.790037    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:50.094647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:50.094611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:50.307898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:50.307843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:52.165022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:52.164982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:52.376596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:52.376561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:55.132714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:55.132660    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:59.678266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:59.678235    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:10:59.681217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:10:59.681200    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qdpw6"
Oct 31 04:11:04.130480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.130450    6931 scope.go:122] "RemoveContainer" containerID="15a28ac42fa5b0e9dea504db790a499f8ab8576749caf0dc77f1346d991cb6aa"
Oct 31 04:11:04.138415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.138398    6931 scope.go:122] "RemoveContainer" containerID="1ac0225044e6ff6de1bfff7985284a23742883b88d4b966714ac08b1765e47fd"
Oct 31 04:11:04.145512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.145493    6931 scope.go:122] "RemoveContainer" containerID="5dce22cace3775452903aab027722ae61094dadea523e6371016b7871438707c"
Oct 31 04:11:04.152786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.152769    6931 scope.go:122] "RemoveContainer" containerID="285d8109a3ac4b8d5a9369971a58f7972d71c9cc5862ca4350835cc91c7081f9"
Oct 31 04:11:04.160772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.160750    6931 scope.go:122] "RemoveContainer" containerID="8afcd2d7211725b74f44efe040584c3b39449208d0c2ef1f6e98101551db0628"
Oct 31 04:11:04.168645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.168625    6931 scope.go:122] "RemoveContainer" containerID="4f0a53c703af8e5bedcf012c45e85e9f67290e96efc17724250ce645533e4f25"
Oct 31 04:11:04.176647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:04.176627    6931 scope.go:122] "RemoveContainer" containerID="eb0af5a40286d72459810c7bd19b4357a25fc11b71d96369fbabfd6433e26ed1"
Oct 31 04:11:10.758341 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:11:10.758308    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:11:22.341516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:22.341459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:22.545574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:22.545535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:23.894106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:23.894077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:24.103534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.103492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:24.301366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.301331    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:24.305266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.305242    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.331307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.331279    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 04:11:24.331456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.331417    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\""
Oct 31 04:11:24.333863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.333838    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:24.415704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.415670    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.415704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.415707    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.415931 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.415738    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.415931 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.415907    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkscv\" (UniqueName: \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.422727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.422697    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:11:24.516875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.516841    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.516875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.516878    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.517137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.516918    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.517137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.516983    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkscv\" (UniqueName: \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.517640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.517617    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.519538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.519517    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.519626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.519559    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.539460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.539436    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkscv\" (UniqueName: \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\") pod \"maas-api-5588f87c7d-gswlv\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") " pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.618544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.618462    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:24.916044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.915957    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:24.921122 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:11:24.921086    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1edc187_2485_497c_bf92_09849c13025a.slice/crio-a49d0ebf3dbd1c0c6dfa185c2e933ea199a81ba957128cebd0d578ed206883a1 WatchSource:0}: Error finding container a49d0ebf3dbd1c0c6dfa185c2e933ea199a81ba957128cebd0d578ed206883a1: Status 404 returned error can't find the container with id a49d0ebf3dbd1c0c6dfa185c2e933ea199a81ba957128cebd0d578ed206883a1
Oct 31 04:11:24.954907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:24.954879    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" event={"ID":"e1edc187-2485-497c-bf92-09849c13025a","Type":"ContainerStarted","Data":"a49d0ebf3dbd1c0c6dfa185c2e933ea199a81ba957128cebd0d578ed206883a1"}
Oct 31 04:11:27.972661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:27.972626    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" event={"ID":"e1edc187-2485-497c-bf92-09849c13025a","Type":"ContainerStarted","Data":"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"}
Oct 31 04:11:27.973062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:27.972735    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:28.014278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:28.014219    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" podStartSLOduration=1.555353814 podStartE2EDuration="4.014206058s" totalImagesPullingTime="2.458852244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:28.013556532 +0000 UTC m=+384.046556268" watchObservedRunningTime="2026-10-31 04:11:28.014206058 +0000 UTC m=+384.047205736"
Oct 31 04:11:33.983545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:33.983513    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:34.596295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:34.596263    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:34.817201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:34.817165    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:34.871711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:34.871635    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9wgm4"]
Oct 31 04:11:34.875535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:34.875514    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:34.892813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:34.892785    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9wgm4"]
Oct 31 04:11:35.018085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.018052    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszbq\" (UniqueName: \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\") pod \"authorino-6f6944bcdd-9wgm4\" (UID: \"25538655-f72f-4a9c-9091-315ff069b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:35.024921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.024890    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9wgm4"]
Oct 31 04:11:35.025116 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:11:35.025098    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zszbq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-9wgm4" podUID="25538655-f72f-4a9c-9091-315ff069b90e"
Oct 31 04:11:35.053790 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.053757    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:35.057470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.057450    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.076425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.076394    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 04:11:35.078067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.078043    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:35.119525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.119495    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.119692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.119543    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpl7g\" (UniqueName: \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.119692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.119656    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszbq\" (UniqueName: \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\") pod \"authorino-6f6944bcdd-9wgm4\" (UID: \"25538655-f72f-4a9c-9091-315ff069b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:35.144998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.144922    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszbq\" (UniqueName: \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\") pod \"authorino-6f6944bcdd-9wgm4\" (UID: \"25538655-f72f-4a9c-9091-315ff069b90e\") " pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:35.221073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.221032    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.221279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.221109    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpl7g\" (UniqueName: \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.223496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.223471    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.244097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.244069    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpl7g\" (UniqueName: \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\") pod \"authorino-6889cbd8c4-lthv8\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") " pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.284166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.284132    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:35.284385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.284373    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:35.585619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:35.585582    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:35.590116 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:11:35.590089    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54a6d2db_c347_4538_b266_b6faafdfc1cc.slice/crio-e37fd759e52b08aa14be02bf193d4587aed74cb3bdad4b6c341de9f179c92950 WatchSource:0}: Error finding container e37fd759e52b08aa14be02bf193d4587aed74cb3bdad4b6c341de9f179c92950: Status 404 returned error can't find the container with id e37fd759e52b08aa14be02bf193d4587aed74cb3bdad4b6c341de9f179c92950
Oct 31 04:11:36.016468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.016377    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" event={"ID":"54a6d2db-c347-4538-b266-b6faafdfc1cc","Type":"ContainerStarted","Data":"e37fd759e52b08aa14be02bf193d4587aed74cb3bdad4b6c341de9f179c92950"}
Oct 31 04:11:36.016468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.016417    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:36.024832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.024807    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:36.128647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.128620    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\" (UniqueName: \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\") pod \"25538655-f72f-4a9c-9091-315ff069b90e\" (UID: \"25538655-f72f-4a9c-9091-315ff069b90e\") "
Oct 31 04:11:36.130621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.130592    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq" pod "25538655-f72f-4a9c-9091-315ff069b90e" (UID: "25538655-f72f-4a9c-9091-315ff069b90e"). InnerVolumeSpecName "kube-api-access-zszbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:36.229463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:36.229426    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszbq\" (UniqueName: \"kubernetes.io/projected/25538655-f72f-4a9c-9091-315ff069b90e-kube-api-access-zszbq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:37.022710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.022670    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" event={"ID":"54a6d2db-c347-4538-b266-b6faafdfc1cc","Type":"ContainerStarted","Data":"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"}
Oct 31 04:11:37.022925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.022705    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" podUID="54a6d2db-c347-4538-b266-b6faafdfc1cc" containerName="authorino" containerID="cri-o://28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c" gracePeriod=30
Oct 31 04:11:37.022925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.022802    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9wgm4"
Oct 31 04:11:37.061475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.061429    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" podStartSLOduration=1.595449787 podStartE2EDuration="2.061416649s" totalImagesPullingTime="465.966862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:37.061030376 +0000 UTC m=+393.094030070" watchObservedRunningTime="2026-10-31 04:11:37.061416649 +0000 UTC m=+393.094416327"
Oct 31 04:11:37.118919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.118882    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9wgm4"]
Oct 31 04:11:37.127570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.127539    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9wgm4"]
Oct 31 04:11:37.266456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.266431    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:37.341828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.341797    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\" (UniqueName: \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\") pod \"54a6d2db-c347-4538-b266-b6faafdfc1cc\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") "
Oct 31 04:11:37.341828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.341831    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\" (UniqueName: \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\") pod \"54a6d2db-c347-4538-b266-b6faafdfc1cc\" (UID: \"54a6d2db-c347-4538-b266-b6faafdfc1cc\") "
Oct 31 04:11:37.343818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.343792    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g" pod "54a6d2db-c347-4538-b266-b6faafdfc1cc" (UID: "54a6d2db-c347-4538-b266-b6faafdfc1cc"). InnerVolumeSpecName "kube-api-access-vpl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:37.352244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.352222    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert" pod "54a6d2db-c347-4538-b266-b6faafdfc1cc" (UID: "54a6d2db-c347-4538-b266-b6faafdfc1cc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:37.443102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.443065    6931 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54a6d2db-c347-4538-b266-b6faafdfc1cc-tls-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:37.443102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:37.443094    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpl7g\" (UniqueName: \"kubernetes.io/projected/54a6d2db-c347-4538-b266-b6faafdfc1cc-kube-api-access-vpl7g\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:38.028947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.028911    6931 generic.go:356] "Generic (PLEG): container finished" podID="54a6d2db-c347-4538-b266-b6faafdfc1cc" containerID="28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c" exitCode=0
Oct 31 04:11:38.029185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.028956    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-lthv8"
Oct 31 04:11:38.029185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.028959    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" event={"ID":"54a6d2db-c347-4538-b266-b6faafdfc1cc","Type":"ContainerDied","Data":"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"}
Oct 31 04:11:38.029185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.029096    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-lthv8" event={"ID":"54a6d2db-c347-4538-b266-b6faafdfc1cc","Type":"ContainerDied","Data":"e37fd759e52b08aa14be02bf193d4587aed74cb3bdad4b6c341de9f179c92950"}
Oct 31 04:11:38.029185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.029127    6931 scope.go:122] "RemoveContainer" containerID="28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"
Oct 31 04:11:38.040712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.040688    6931 scope.go:122] "RemoveContainer" containerID="28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"
Oct 31 04:11:38.041260 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:11:38.041235    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c\": container with ID starting with 28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c not found: ID does not exist" containerID="28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"
Oct 31 04:11:38.041352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.041269    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c"} err="failed to get container status \"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c\": rpc error: code = NotFound desc = could not find container \"28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c\": container with ID starting with 28e1cc7f1e4dbe993df50e086caefbfb5fe750c0cdc60c3679be9c385e49aa8c not found: ID does not exist"
Oct 31 04:11:38.078003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.077967    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:38.089866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.089836    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-lthv8"]
Oct 31 04:11:38.120711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.120680    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25538655-f72f-4a9c-9091-315ff069b90e" path="/var/lib/kubelet/pods/25538655-f72f-4a9c-9091-315ff069b90e/volumes"
Oct 31 04:11:38.120936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.120923    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a6d2db-c347-4538-b266-b6faafdfc1cc" path="/var/lib/kubelet/pods/54a6d2db-c347-4538-b266-b6faafdfc1cc/volumes"
Oct 31 04:11:38.816443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.816393    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:11:38.816646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:38.816609    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" podUID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" containerName="authorino" containerID="cri-o://4fc7ceec1ae33974b80f899bc6a02dd26d3140e3c863f75b68b1004e89fbc52e" gracePeriod=30
Oct 31 04:11:39.036205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.036147    6931 generic.go:356] "Generic (PLEG): container finished" podID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" containerID="4fc7ceec1ae33974b80f899bc6a02dd26d3140e3c863f75b68b1004e89fbc52e" exitCode=0
Oct 31 04:11:39.036205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.036203    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" event={"ID":"26b755b6-314e-4022-9c5c-3f55d68d3f3a","Type":"ContainerDied","Data":"4fc7ceec1ae33974b80f899bc6a02dd26d3140e3c863f75b68b1004e89fbc52e"}
Oct 31 04:11:39.067078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.067020    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:11:39.156991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.156956    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\") pod \"26b755b6-314e-4022-9c5c-3f55d68d3f3a\" (UID: \"26b755b6-314e-4022-9c5c-3f55d68d3f3a\") "
Oct 31 04:11:39.159025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.159003    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt" pod "26b755b6-314e-4022-9c5c-3f55d68d3f3a" (UID: "26b755b6-314e-4022-9c5c-3f55d68d3f3a"). InnerVolumeSpecName "kube-api-access-wm4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:39.258772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.258714    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/26b755b6-314e-4022-9c5c-3f55d68d3f3a-kube-api-access-wm4nt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:39.882976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.882940    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"]
Oct 31 04:11:39.883380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883365    6931 cpu_manager.go:414] "containerMap: removing container" podUID="54a6d2db-c347-4538-b266-b6faafdfc1cc" containerName="authorino"
Oct 31 04:11:39.883433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883381    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a6d2db-c347-4538-b266-b6faafdfc1cc" containerName="authorino"
Oct 31 04:11:39.883433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883397    6931 cpu_manager.go:414] "containerMap: removing container" podUID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" containerName="authorino"
Oct 31 04:11:39.883433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883402    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" containerName="authorino"
Oct 31 04:11:39.883522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883482    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a6d2db-c347-4538-b266-b6faafdfc1cc" containerName="authorino"
Oct 31 04:11:39.883522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.883491    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" containerName="authorino"
Oct 31 04:11:39.886688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.886671    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:39.905576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.905539    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"]
Oct 31 04:11:39.963633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.963592    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4dbbfa-848a-439f-bef4-172b3ea50d73-cert\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:39.963802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:39.963643    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5q2\" (UniqueName: \"kubernetes.io/projected/9d4dbbfa-848a-439f-bef4-172b3ea50d73-kube-api-access-vf5q2\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.043562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.043529    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq" event={"ID":"26b755b6-314e-4022-9c5c-3f55d68d3f3a","Type":"ContainerDied","Data":"5081c8ef7f765656218f8c3b9ab1a0fe1649c8993e614e0908ced6add5cb9e6d"}
Oct 31 04:11:40.043722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.043566    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgrwq"
Oct 31 04:11:40.043722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.043577    6931 scope.go:122] "RemoveContainer" containerID="4fc7ceec1ae33974b80f899bc6a02dd26d3140e3c863f75b68b1004e89fbc52e"
Oct 31 04:11:40.064739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.064708    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4dbbfa-848a-439f-bef4-172b3ea50d73-cert\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.064861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.064767    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5q2\" (UniqueName: \"kubernetes.io/projected/9d4dbbfa-848a-439f-bef4-172b3ea50d73-kube-api-access-vf5q2\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.067505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.067485    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4dbbfa-848a-439f-bef4-172b3ea50d73-cert\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.088248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.088221    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5q2\" (UniqueName: \"kubernetes.io/projected/9d4dbbfa-848a-439f-bef4-172b3ea50d73-kube-api-access-vf5q2\") pod \"llmisvc-controller-manager-6466c554d8-8b55h\" (UID: \"9d4dbbfa-848a-439f-bef4-172b3ea50d73\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.092880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.092849    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:11:40.097410 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.097382    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgrwq"]
Oct 31 04:11:40.122403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.122371    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b755b6-314e-4022-9c5c-3f55d68d3f3a" path="/var/lib/kubelet/pods/26b755b6-314e-4022-9c5c-3f55d68d3f3a/volumes"
Oct 31 04:11:40.198815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.198724    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:40.499252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:40.499109    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"]
Oct 31 04:11:40.504087 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:11:40.504058    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4dbbfa_848a_439f_bef4_172b3ea50d73.slice/crio-ea4cc55beec8ba653628cbe02db4d131437012c488dd92ce53407216901e5e5b WatchSource:0}: Error finding container ea4cc55beec8ba653628cbe02db4d131437012c488dd92ce53407216901e5e5b: Status 404 returned error can't find the container with id ea4cc55beec8ba653628cbe02db4d131437012c488dd92ce53407216901e5e5b
Oct 31 04:11:40.758633 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:11:40.758555    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:11:41.049683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:41.049641    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h" event={"ID":"9d4dbbfa-848a-439f-bef4-172b3ea50d73","Type":"ContainerStarted","Data":"ce54bcd95d35d117f91c9dcd7ed2a1b96ba1d2d73f680c01a8303489fdb843c8"}
Oct 31 04:11:41.049868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:41.049696    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h" event={"ID":"9d4dbbfa-848a-439f-bef4-172b3ea50d73","Type":"ContainerStarted","Data":"ea4cc55beec8ba653628cbe02db4d131437012c488dd92ce53407216901e5e5b"}
Oct 31 04:11:41.049868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:41.049771    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:11:41.090362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:41.090309    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h" podStartSLOduration=1.6987510540000001 podStartE2EDuration="2.090296304s" totalImagesPullingTime="391.54525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:41.08936713 +0000 UTC m=+397.122366830" watchObservedRunningTime="2026-10-31 04:11:41.090296304 +0000 UTC m=+397.123295984"
Oct 31 04:11:44.215804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.215771    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:44.216263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.216013    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" podUID="e1edc187-2485-497c-bf92-09849c13025a" containerName="maas-api" containerID="cri-o://9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb" gracePeriod=30
Oct 31 04:11:44.460894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.460871    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:44.607758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.607726    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\") pod \"e1edc187-2485-497c-bf92-09849c13025a\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") "
Oct 31 04:11:44.607969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.607784    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\" (UniqueName: \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\") pod \"e1edc187-2485-497c-bf92-09849c13025a\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") "
Oct 31 04:11:44.607969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.607833    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\") pod \"e1edc187-2485-497c-bf92-09849c13025a\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") "
Oct 31 04:11:44.608110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.608030    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\") pod \"e1edc187-2485-497c-bf92-09849c13025a\" (UID: \"e1edc187-2485-497c-bf92-09849c13025a\") "
Oct 31 04:11:44.608260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.608231    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca" pod "e1edc187-2485-497c-bf92-09849c13025a" (UID: "e1edc187-2485-497c-bf92-09849c13025a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:44.608383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.608343    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1edc187-2485-497c-bf92-09849c13025a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:44.610047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.610022    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs" pod "e1edc187-2485-497c-bf92-09849c13025a" (UID: "e1edc187-2485-497c-bf92-09849c13025a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:44.610142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.610060    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls" pod "e1edc187-2485-497c-bf92-09849c13025a" (UID: "e1edc187-2485-497c-bf92-09849c13025a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:44.610142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.610090    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv" pod "e1edc187-2485-497c-bf92-09849c13025a" (UID: "e1edc187-2485-497c-bf92-09849c13025a"). InnerVolumeSpecName "kube-api-access-lkscv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:44.709663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.709626    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:44.709663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.709654    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1edc187-2485-497c-bf92-09849c13025a-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:44.709663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:44.709664    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkscv\" (UniqueName: \"kubernetes.io/projected/e1edc187-2485-497c-bf92-09849c13025a-kube-api-access-lkscv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:11:45.072238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.072205    6931 generic.go:356] "Generic (PLEG): container finished" podID="e1edc187-2485-497c-bf92-09849c13025a" containerID="9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb" exitCode=0
Oct 31 04:11:45.072400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.072253    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"
Oct 31 04:11:45.072400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.072265    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" event={"ID":"e1edc187-2485-497c-bf92-09849c13025a","Type":"ContainerDied","Data":"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"}
Oct 31 04:11:45.072400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.072308    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv" event={"ID":"e1edc187-2485-497c-bf92-09849c13025a","Type":"ContainerDied","Data":"a49d0ebf3dbd1c0c6dfa185c2e933ea199a81ba957128cebd0d578ed206883a1"}
Oct 31 04:11:45.072400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.072329    6931 scope.go:122] "RemoveContainer" containerID="9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"
Oct 31 04:11:45.081921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.081903    6931 scope.go:122] "RemoveContainer" containerID="9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"
Oct 31 04:11:45.082221 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:11:45.082201    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb\": container with ID starting with 9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb not found: ID does not exist" containerID="9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"
Oct 31 04:11:45.082302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.082230    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb"} err="failed to get container status \"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb\": rpc error: code = NotFound desc = could not find container \"9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb\": container with ID starting with 9116dc40c3a3b7769aa070573e2bd924579bc96e97f80314838851ec724c50eb not found: ID does not exist"
Oct 31 04:11:45.123203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.123167    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:45.135103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:45.135074    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5588f87c7d-gswlv"]
Oct 31 04:11:46.120826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:46.120795    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1edc187-2485-497c-bf92-09849c13025a" path="/var/lib/kubelet/pods/e1edc187-2485-497c-bf92-09849c13025a/volumes"
Oct 31 04:11:55.253837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:11:55.253799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:10.758436 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:12:10.758393    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:12:12.057446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.057408    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8b55h"
Oct 31 04:12:12.129755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.129716    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:12:12.129975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.129930    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" podUID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" containerName="manager" containerID="cri-o://747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c" gracePeriod=30
Oct 31 04:12:12.379046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.379020    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:12:12.467922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.467884    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\" (UniqueName: \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\") pod \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") "
Oct 31 04:12:12.468113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.467985    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\" (UniqueName: \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\") pod \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\" (UID: \"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778\") "
Oct 31 04:12:12.470059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.470032    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn" pod "63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" (UID: "63dfb6a0-5f2e-4744-a8c7-a3d307fe2778"). InnerVolumeSpecName "kube-api-access-6j7pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:12:12.470187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.470145    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert" pod "63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" (UID: "63dfb6a0-5f2e-4744-a8c7-a3d307fe2778"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:12:12.569355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.569316    6931 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:12:12.569355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:12.569350    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7pn\" (UniqueName: \"kubernetes.io/projected/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778-kube-api-access-6j7pn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:12:13.201519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.201485    6931 generic.go:356] "Generic (PLEG): container finished" podID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" containerID="747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c" exitCode=0
Oct 31 04:12:13.201943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.201529    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"
Oct 31 04:12:13.201943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.201558    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" event={"ID":"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778","Type":"ContainerDied","Data":"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"}
Oct 31 04:12:13.201943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.201611    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd" event={"ID":"63dfb6a0-5f2e-4744-a8c7-a3d307fe2778","Type":"ContainerDied","Data":"ebe30795b03b6a28bc71ecb70a2e87c9295b700832ab62c08c8918ac62892ddd"}
Oct 31 04:12:13.201943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.201634    6931 scope.go:122] "RemoveContainer" containerID="747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"
Oct 31 04:12:13.210464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.210444    6931 scope.go:122] "RemoveContainer" containerID="747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"
Oct 31 04:12:13.210735 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:12:13.210717    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c\": container with ID starting with 747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c not found: ID does not exist" containerID="747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"
Oct 31 04:12:13.210779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.210744    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c"} err="failed to get container status \"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c\": rpc error: code = NotFound desc = could not find container \"747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c\": container with ID starting with 747a024460214b0035a4158e0a54f7e6103e46a1229e1c88b963ce9d210af25c not found: ID does not exist"
Oct 31 04:12:13.251491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.251456    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:12:13.263320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.263289    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wrscd"]
Oct 31 04:12:13.929889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:13.929857    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:14.122082 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:14.122050    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" path="/var/lib/kubelet/pods/63dfb6a0-5f2e-4744-a8c7-a3d307fe2778/volumes"
Oct 31 04:12:14.135737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:14.135708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:18.226675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:18.226634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:18.439747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:18.439693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:20.666229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:20.666179    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:28.256770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:28.256723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:28.471099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:28.471064    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:40.758712 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:12:40.758671    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:12:41.638261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:41.638218    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:41.863909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:41.863873    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:44.016613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:44.016575    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:44.593773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:44.593732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:46.022110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:46.022069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:46.242473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:46.242428    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:50.247522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:50.247484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:53.671137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:53.671086    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:55.385984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:55.385933    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:55.603561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:55.603524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:57.187712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:57.187676    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:59.554732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:12:59.554690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:00.670829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:00.670787    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:00.883058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:00.883013    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:01.875545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:01.875499    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:03.568755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:03.568720    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:10.758271 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:13:10.758186    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:13:23.552407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.552365    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:23.710463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.710428    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"]
Oct 31 04:13:23.711135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711104    6931 cpu_manager.go:414] "containerMap: removing container" podUID="e1edc187-2485-497c-bf92-09849c13025a" containerName="maas-api"
Oct 31 04:13:23.711135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711134    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1edc187-2485-497c-bf92-09849c13025a" containerName="maas-api"
Oct 31 04:13:23.711363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711150    6931 cpu_manager.go:414] "containerMap: removing container" podUID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" containerName="manager"
Oct 31 04:13:23.711363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711184    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" containerName="manager"
Oct 31 04:13:23.711363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711262    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1edc187-2485-497c-bf92-09849c13025a" containerName="maas-api"
Oct 31 04:13:23.711363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.711274    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="63dfb6a0-5f2e-4744-a8c7-a3d307fe2778" containerName="manager"
Oct 31 04:13:23.714644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.714625    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.730720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.730694    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 04:13:23.744343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.744308    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"]
Oct 31 04:13:23.751792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.751768    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sqqs6\""
Oct 31 04:13:23.777545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.777509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:23.897878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.897781    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.897878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.897830    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.897895    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.897947    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjql\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-kube-api-access-sgjql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.897990    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.898012    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.898056    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e490550-c470-4717-9185-798d55ef73b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.898198    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e490550-c470-4717-9185-798d55ef73b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.898284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.898244    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.998917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.998877    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.998932    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.998955    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.998983    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999005    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjql\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-kube-api-access-sgjql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999050    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999081    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999109    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e490550-c470-4717-9185-798d55ef73b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999316    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e490550-c470-4717-9185-798d55ef73b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999364    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999399    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999547    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999649    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:23.999823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:23.999796    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e490550-c470-4717-9185-798d55ef73b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.001576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.001550    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e490550-c470-4717-9185-798d55ef73b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.001797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.001776    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e490550-c470-4717-9185-798d55ef73b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.024932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.024899    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.025057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.025040    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjql\" (UniqueName: \"kubernetes.io/projected/9e490550-c470-4717-9185-798d55ef73b2-kube-api-access-sgjql\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wzfdx\" (UID: \"9e490550-c470-4717-9185-798d55ef73b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.028924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.028900    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:24.347310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.347272    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"]
Oct 31 04:13:24.353407 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:13:24.353379    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e490550_c470_4717_9185_798d55ef73b2.slice/crio-fd6fbf756b48347c3619cc0d2326b85899ac9b509efec83190f2bc70633ba5aa WatchSource:0}: Error finding container fd6fbf756b48347c3619cc0d2326b85899ac9b509efec83190f2bc70633ba5aa: Status 404 returned error can't find the container with id fd6fbf756b48347c3619cc0d2326b85899ac9b509efec83190f2bc70633ba5aa
Oct 31 04:13:24.355282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.355266    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:13:24.549251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:24.549215    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" event={"ID":"9e490550-c470-4717-9185-798d55ef73b2","Type":"ContainerStarted","Data":"fd6fbf756b48347c3619cc0d2326b85899ac9b509efec83190f2bc70633ba5aa"}
Oct 31 04:13:25.213649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:25.213605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:25.534748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:25.534714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:26.944820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:26.944784    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:26.945107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:26.944854    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:26.945107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:26.944882    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:27.566828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:27.566794    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" event={"ID":"9e490550-c470-4717-9185-798d55ef73b2","Type":"ContainerStarted","Data":"b1a4371820a77e43b082c2b033c508ecbddcf74cadbd606915ea5002b2bd3dfd"}
Oct 31 04:13:27.623928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:27.623875    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podStartSLOduration=2.035348947 podStartE2EDuration="4.623861938s" totalImagesPullingTime="2.588512991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:27.623424469 +0000 UTC m=+503.656424168" watchObservedRunningTime="2026-10-31 04:13:27.623861938 +0000 UTC m=+503.656861617"
Oct 31 04:13:28.040061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:28.039972    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:28.040720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:28.040692    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:28.040906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:28.040880    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:29.033011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:29.032546    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:29.033011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:29.032607    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:30.042183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:30.030611    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:30.042183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:30.030678    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:31.051658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:31.051609    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:31.052233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:31.051685    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:32.037645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:32.037602    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:32.037965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:32.037931    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:33.029533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:33.029496    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:33.029952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:33.029554    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:34.030087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:34.030047    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 04:13:34.030525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:34.030101    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 04:13:35.130869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:35.130836    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:35.131597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:35.131254    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:35.132184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:35.132146    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx"
Oct 31 04:13:38.759016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.758985    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:13:38.762967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.762944    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.787618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.787482    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:13:38.856720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856681    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.856895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856725    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.856895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856800    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.856895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856841    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.857002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856894    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.857085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.856993    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.857085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.857040    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.857148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.857104    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.857148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.857127    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwc7\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.957715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957666    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.957715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957720    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.957920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957815    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwc7\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.957920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957876    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.957920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957902    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957923    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.957969    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958002    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958033    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958476 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958445    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958476 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958473    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958532    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958599    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.958896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.958874    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.960115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.960093    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.960349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.960331    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.980765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.980739    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:38.980863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:38.980843    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwc7\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zw5nk\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:39.078515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.078484    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:39.395881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.395790    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:13:39.400968 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:13:39.400933    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532f15ed_92bd_4c7c_92f9_bb0ffc3af80a.slice/crio-1632445bd1c21760a6751519d5aefc1f25695d7c544438560552899c04dac267 WatchSource:0}: Error finding container 1632445bd1c21760a6751519d5aefc1f25695d7c544438560552899c04dac267: Status 404 returned error can't find the container with id 1632445bd1c21760a6751519d5aefc1f25695d7c544438560552899c04dac267
Oct 31 04:13:39.403014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.402980    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:39.403105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.403060    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:39.403202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.403106    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:13:39.625750 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.625704    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" event={"ID":"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a","Type":"ContainerStarted","Data":"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"}
Oct 31 04:13:39.625917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.625775    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" event={"ID":"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a","Type":"ContainerStarted","Data":"1632445bd1c21760a6751519d5aefc1f25695d7c544438560552899c04dac267"}
Oct 31 04:13:39.671313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:39.671205    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podStartSLOduration=1.6711518330000001 podStartE2EDuration="1.671151833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:39.670057495 +0000 UTC m=+515.703057198" watchObservedRunningTime="2026-10-31 04:13:39.671151833 +0000 UTC m=+515.704151513"
Oct 31 04:13:40.079536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:40.079499    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:40.080878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:40.080853    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:40.080988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:40.080916    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:40.772950 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:13:40.772914    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:13:41.079490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:41.079445    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:41.079653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:41.079518    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:42.079113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:42.079073    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:42.079289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:42.079133    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:43.079461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:43.079427    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:43.079835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:43.079508    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:44.079113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:44.079074    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:44.079321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:44.079141    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:45.079665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:45.079630    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:45.080040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:45.079696    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:46.079480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:46.079439    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 31 04:13:46.079688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:46.079494    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 31 04:13:46.209350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:46.209307    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:47.630199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:47.630145    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:47.630983 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:47.630961    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:47.631693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:47.631673    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:13:47.840594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:47.840553    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:54.167195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:54.167112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:54.395287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:54.395254    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:54.608607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:54.608570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:54.937330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:54.937242    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:55.148232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:55.148193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:55.361368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:55.361329    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:55.575029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:55.574997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:56.389628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:56.389590    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:56.916200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:56.916146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:57.129823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:57.129786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:57.503937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:57.503891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:57.713099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:57.713065    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:59.179654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.179619    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:13:59.183422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.183401    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.211464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.211437    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:13:59.211464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.211447    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-16cda44a-serving-cert\""
Oct 31 04:13:59.223317 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.223274    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\""
Oct 31 04:13:59.224949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.224918    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:13:59.256429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.256396    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.256603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.256440    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.256674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.256631    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmx2w\" (UniqueName: \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.256737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.256704    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.261271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.261241    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:59.358019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.357986    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmx2w\" (UniqueName: \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.358244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.358032    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.358244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.358077    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.358244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.358096    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.358807 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.358777    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.360629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.360608    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.360725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.360650    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.388747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.388716    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmx2w\" (UniqueName: \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\") pod \"maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.473190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.473093    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:59.496748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.496719    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:13:59.700385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.700348    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:59.806465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.806425    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:13:59.811225 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:13:59.811199    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebee838b_8183_47cd_86b2_b4cee0465c56.slice/crio-40cb4a92e12496727cd6c000f7f3d7371923e949302af747a3ec34ff2166b01f WatchSource:0}: Error finding container 40cb4a92e12496727cd6c000f7f3d7371923e949302af747a3ec34ff2166b01f: Status 404 returned error can't find the container with id 40cb4a92e12496727cd6c000f7f3d7371923e949302af747a3ec34ff2166b01f
Oct 31 04:13:59.968623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:13:59.968587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:00.202270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.202145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:00.753307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.753224    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" event={"ID":"ebee838b-8183-47cd-86b2-b4cee0465c56","Type":"ContainerStarted","Data":"da138380a89f15956399b9e4c7e27e8cee54c0d3567b6d707ca8616e79fe2daa"}
Oct 31 04:14:00.753307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.753280    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" event={"ID":"ebee838b-8183-47cd-86b2-b4cee0465c56","Type":"ContainerStarted","Data":"40cb4a92e12496727cd6c000f7f3d7371923e949302af747a3ec34ff2166b01f"}
Oct 31 04:14:00.753497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.753343    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:14:00.797845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.797604    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" podStartSLOduration=1.1972758510000001 podStartE2EDuration="1.797588989s" totalImagesPullingTime="600.313138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:00.79692808 +0000 UTC m=+536.829927777" watchObservedRunningTime="2026-10-31 04:14:00.797588989 +0000 UTC m=+536.830588667"
Oct 31 04:14:00.991942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:00.991910    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:01.214584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.214535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:01.278765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.278729    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:01.281658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.281635    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.303702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.303672    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:01.304828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.304799    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a0d1a61d-serving-cert\""
Oct 31 04:14:01.377381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.377346    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.377551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.377401    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.377551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.377475    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.377551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.377506    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.478359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.478268    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.478359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.478310    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.478359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.478344    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.478658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.478424    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.479082 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.479053    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.480722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.480700    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.480828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.480811    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.486200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.486169    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:01.501064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.501036    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\") pod \"maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.593456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.593421    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:01.696861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.696817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:01.892591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:01.892556    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:01.897419 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:01.897393    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41f28d0_2a31_4058_8bde_d08165d89340.slice/crio-4d28b889ed6aac86ca61c2703f15bd441b02c89148ef66034c57a1695f4c5b71 WatchSource:0}: Error finding container 4d28b889ed6aac86ca61c2703f15bd441b02c89148ef66034c57a1695f4c5b71: Status 404 returned error can't find the container with id 4d28b889ed6aac86ca61c2703f15bd441b02c89148ef66034c57a1695f4c5b71
Oct 31 04:14:02.222226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.222195    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:02.503051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.502968    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:02.711910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.711872    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:02.765030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.764952    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" event={"ID":"a41f28d0-2a31-4058-8bde-d08165d89340","Type":"ContainerStarted","Data":"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"}
Oct 31 04:14:02.765030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.765008    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" event={"ID":"a41f28d0-2a31-4058-8bde-d08165d89340","Type":"ContainerStarted","Data":"4d28b889ed6aac86ca61c2703f15bd441b02c89148ef66034c57a1695f4c5b71"}
Oct 31 04:14:02.765291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.765093    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:02.811431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:02.811369    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" podStartSLOduration=1.5174953310000001 podStartE2EDuration="1.811354503s" totalImagesPullingTime="293.859172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:02.810945684 +0000 UTC m=+538.843945391" watchObservedRunningTime="2026-10-31 04:14:02.811354503 +0000 UTC m=+538.844354181"
Oct 31 04:14:03.149121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:03.149085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:03.400231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:03.400106    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:03.618288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:03.618251    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:04.281353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.281314    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:04.305737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.305705    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:04.309083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.309063    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.323605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.323575    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:04.333344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.333321    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4805aacc\""
Oct 31 04:14:04.333498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.333388    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4805aacc-dockercfg-h4gwn\""
Oct 31 04:14:04.410204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.410148    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj97\" (UniqueName: \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.410703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.410368    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.506104 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.506077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:04.511299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.511270    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.511412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.511308    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj97\" (UniqueName: \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.512377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.512359    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.535179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.535098    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj97\" (UniqueName: \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\") pod \"payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") " pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.621473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.621438    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:04.920699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:04.920614    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:04.924952 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:04.924924    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366b873_e392_4c6c_b5ce_b60b3daed546.slice/crio-63a93be975c7cdf9742f036b410f1ac3bb26605723c9a8a52d2f3fef7c4d6c24 WatchSource:0}: Error finding container 63a93be975c7cdf9742f036b410f1ac3bb26605723c9a8a52d2f3fef7c4d6c24: Status 404 returned error can't find the container with id 63a93be975c7cdf9742f036b410f1ac3bb26605723c9a8a52d2f3fef7c4d6c24
Oct 31 04:14:05.251013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.250899    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:05.255752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.255730    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.277104 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.277078    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4805aacc-serving-cert\""
Oct 31 04:14:05.279342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.279312    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:05.419532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.419497    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcpn\" (UniqueName: \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.419921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.419544    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.419921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.419686    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.419921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.419771    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.462623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.462583    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:05.521428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.521342    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcpn\" (UniqueName: \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.521428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.521388    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.521679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.521453    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.521679 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:05.521602    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-serving-cert: secret "maas-api-e2e-ait-4805aacc-serving-cert" not found
Oct 31 04:14:05.521679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.521621    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.521814 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:05.521692    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls podName:1545d84f-898c-4831-bc2c-40fbb79153e7 nodeName:}" failed. No retries permitted until 2026-10-31 04:14:06.021654449 +0000 UTC m=+542.054654114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls") pod "maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" (UID: "1545d84f-898c-4831-bc2c-40fbb79153e7") : secret "maas-api-e2e-ait-4805aacc-serving-cert" not found
Oct 31 04:14:05.522366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.522341    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.525051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.525003    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.547204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.547174    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcpn\" (UniqueName: \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:05.684723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.684683    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:05.783751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.783718    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" event={"ID":"7366b873-e392-4c6c-b5ce-b60b3daed546","Type":"ContainerStarted","Data":"63a93be975c7cdf9742f036b410f1ac3bb26605723c9a8a52d2f3fef7c4d6c24"}
Oct 31 04:14:05.897209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:05.897143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:06.028169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.028113    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:06.030991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.030966    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") pod \"maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:06.169407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.169318    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:06.491765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.491674    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:06.496902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:06.496856    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1545d84f_898c_4831_bc2c_40fbb79153e7.slice/crio-49264c9a0b21911df73d53494053725a321864c5b250c0da5f1fff947fc10ecc WatchSource:0}: Error finding container 49264c9a0b21911df73d53494053725a321864c5b250c0da5f1fff947fc10ecc: Status 404 returned error can't find the container with id 49264c9a0b21911df73d53494053725a321864c5b250c0da5f1fff947fc10ecc
Oct 31 04:14:06.607966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.607935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:06.765315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.765238    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:14:06.789580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.789548    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" event={"ID":"1545d84f-898c-4831-bc2c-40fbb79153e7","Type":"ContainerStarted","Data":"49264c9a0b21911df73d53494053725a321864c5b250c0da5f1fff947fc10ecc"}
Oct 31 04:14:06.838769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:06.838739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:07.674860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.674809    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:07.796848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.796806    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" event={"ID":"7366b873-e392-4c6c-b5ce-b60b3daed546","Type":"ContainerStarted","Data":"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"}
Oct 31 04:14:07.797034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.797000    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:07.798434 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.798409    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" event={"ID":"1545d84f-898c-4831-bc2c-40fbb79153e7","Type":"ContainerStarted","Data":"1e0a7448218ec6374d70f06e6ac27419020cb90f33093fef53d80f8b6b6e990e"}
Oct 31 04:14:07.798548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.798462    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:07.837414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.837355    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" podStartSLOduration=1.905313016 podStartE2EDuration="3.837342207s" totalImagesPullingTime="1.932029191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:07.836337806 +0000 UTC m=+543.869337484" watchObservedRunningTime="2026-10-31 04:14:07.837342207 +0000 UTC m=+543.870341886"
Oct 31 04:14:07.878498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:07.878445    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" podStartSLOduration=2.519401132 podStartE2EDuration="2.87843141s" totalImagesPullingTime="359.030278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:07.877286856 +0000 UTC m=+543.910286534" watchObservedRunningTime="2026-10-31 04:14:07.87843141 +0000 UTC m=+543.911431089"
Oct 31 04:14:08.574140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:08.574102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:08.774541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:08.774514    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:08.791908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:08.791873    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:09.779151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:09.779112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:09.996593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:09.996552    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:10.207532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:10.207498    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:10.758448 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:10.758413    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:14:11.713226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:11.713191    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:13.442526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:13.442485    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:13.809131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:13.809102    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:15.765806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:15.765769    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:15.976808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:15.976777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:18.386774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:18.386733    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:18.806705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:18.806673    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:20.496545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:20.496507    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:22.508947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:22.508895    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:24.265756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:24.265714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:24.477341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:24.477306    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:24.685012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:24.684967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:25.122297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.122265    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:25.126281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.126259    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.140460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.140436    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-44f032ad-openshift-default-dockercfg-qx5tv\""
Oct 31 04:14:25.154711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.154685    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:25.211726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211689    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.211917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211766    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.211917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211803    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.211917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211836    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsglb\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.211917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211900    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.212086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.211957    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.212390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.212215    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.212390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.212311    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.212534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.212402    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.313631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313585    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313640    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313666    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313728    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313785    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313812    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313840    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsglb\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313867    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.313903    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.314286    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.314391    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.314404    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.314483    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.314789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.314693    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.316242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.316220    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.316344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.316292    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.336819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.336789    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.336942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.336920    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsglb\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\") pod \"e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") " pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.440419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.440336    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:25.759683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.759600    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:25.763701 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:25.763672    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8d4620_d98c_41da_979f_539f7846df0e.slice/crio-a871edc19376c59e05751fe562e116be688b642524753d13cb7b40c139e88a9a WatchSource:0}: Error finding container a871edc19376c59e05751fe562e116be688b642524753d13cb7b40c139e88a9a: Status 404 returned error can't find the container with id a871edc19376c59e05751fe562e116be688b642524753d13cb7b40c139e88a9a
Oct 31 04:14:25.765956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.765926    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:14:25.766030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.765985    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:14:25.766030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.766016    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:14:25.899440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.899400    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" event={"ID":"9c8d4620-d98c-41da-979f-539f7846df0e","Type":"ContainerStarted","Data":"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"}
Oct 31 04:14:25.899716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.899687    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" event={"ID":"9c8d4620-d98c-41da-979f-539f7846df0e","Type":"ContainerStarted","Data":"a871edc19376c59e05751fe562e116be688b642524753d13cb7b40c139e88a9a"}
Oct 31 04:14:25.947148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:25.947081    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podStartSLOduration=0.947057912 podStartE2EDuration="947.057912ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:25.94387617 +0000 UTC m=+561.976875875" watchObservedRunningTime="2026-10-31 04:14:25.947057912 +0000 UTC m=+561.980057593"
Oct 31 04:14:26.441211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:26.441140    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:26.442652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:26.442628    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:26.442749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:26.442673    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:26.654737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:26.654702    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:27.448651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:27.448607    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:27.449309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:27.449276    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:28.447180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:28.446606    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:28.447180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:28.446674    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:29.013086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:29.013050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:29.447717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:29.447673    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:29.448000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:29.447963    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:29.796799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:29.796752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:30.005885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:30.005843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:30.451801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:30.451764    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:30.452385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:30.451829    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:31.177870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:31.177834    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:31.441681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:31.441583    6931 patch_prober.go:29] interesting pod/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 04:14:31.441975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:31.441943    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 04:14:32.445077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.445048    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:32.445489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.445481    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:32.446091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.446074    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:32.711306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.711224    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:14:32.711473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.711446    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" podUID="ebee838b-8183-47cd-86b2-b4cee0465c56" containerName="maas-api" containerID="cri-o://da138380a89f15956399b9e4c7e27e8cee54c0d3567b6d707ca8616e79fe2daa" gracePeriod=30
Oct 31 04:14:32.888142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.888092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:32.946741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.946710    6931 generic.go:356] "Generic (PLEG): container finished" podID="ebee838b-8183-47cd-86b2-b4cee0465c56" containerID="da138380a89f15956399b9e4c7e27e8cee54c0d3567b6d707ca8616e79fe2daa" exitCode=0
Oct 31 04:14:32.946928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.946778    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" event={"ID":"ebee838b-8183-47cd-86b2-b4cee0465c56","Type":"ContainerDied","Data":"da138380a89f15956399b9e4c7e27e8cee54c0d3567b6d707ca8616e79fe2daa"}
Oct 31 04:14:32.972279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:32.972211    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:14:33.118978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.118942    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\" (UniqueName: \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\") pod \"ebee838b-8183-47cd-86b2-b4cee0465c56\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") "
Oct 31 04:14:33.119224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.119053    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\") pod \"ebee838b-8183-47cd-86b2-b4cee0465c56\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") "
Oct 31 04:14:33.119224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.119193    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\") pod \"ebee838b-8183-47cd-86b2-b4cee0465c56\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") "
Oct 31 04:14:33.119360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.119235    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\") pod \"ebee838b-8183-47cd-86b2-b4cee0465c56\" (UID: \"ebee838b-8183-47cd-86b2-b4cee0465c56\") "
Oct 31 04:14:33.119664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.119632    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca" pod "ebee838b-8183-47cd-86b2-b4cee0465c56" (UID: "ebee838b-8183-47cd-86b2-b4cee0465c56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:33.121350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.121320    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs" pod "ebee838b-8183-47cd-86b2-b4cee0465c56" (UID: "ebee838b-8183-47cd-86b2-b4cee0465c56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:33.121459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.121409    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls" pod "ebee838b-8183-47cd-86b2-b4cee0465c56" (UID: "ebee838b-8183-47cd-86b2-b4cee0465c56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:33.121459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.121427    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w" pod "ebee838b-8183-47cd-86b2-b4cee0465c56" (UID: "ebee838b-8183-47cd-86b2-b4cee0465c56"). InnerVolumeSpecName "kube-api-access-jmx2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:33.221109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.221060    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:33.221109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.221101    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebee838b-8183-47cd-86b2-b4cee0465c56-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:33.221109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.221116    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmx2w\" (UniqueName: \"kubernetes.io/projected/ebee838b-8183-47cd-86b2-b4cee0465c56-kube-api-access-jmx2w\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:33.221379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.221128    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebee838b-8183-47cd-86b2-b4cee0465c56-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:33.621858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.621824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:33.842484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.842446    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:33.953437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.953359    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"
Oct 31 04:14:33.953437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.953391    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q" event={"ID":"ebee838b-8183-47cd-86b2-b4cee0465c56","Type":"ContainerDied","Data":"40cb4a92e12496727cd6c000f7f3d7371923e949302af747a3ec34ff2166b01f"}
Oct 31 04:14:33.953642 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:33.953457    6931 scope.go:122] "RemoveContainer" containerID="da138380a89f15956399b9e4c7e27e8cee54c0d3567b6d707ca8616e79fe2daa"
Oct 31 04:14:34.006333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:34.006297    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:14:34.018398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:34.018362    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16cda44a-65d585fdd7-m5l9q"]
Oct 31 04:14:34.048506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:34.048468    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:34.121316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:34.121286    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebee838b-8183-47cd-86b2-b4cee0465c56" path="/var/lib/kubelet/pods/ebee838b-8183-47cd-86b2-b4cee0465c56/volumes"
Oct 31 04:14:34.805121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:34.805082    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.316199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.316070    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:35.319527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.316411    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" podUID="a41f28d0-2a31-4058-8bde-d08165d89340" containerName="maas-api" containerID="cri-o://a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86" gracePeriod=30
Oct 31 04:14:35.355496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.355455    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.569785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.569697    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.596059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.596027    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:35.749000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.748959    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\") pod \"a41f28d0-2a31-4058-8bde-d08165d89340\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") "
Oct 31 04:14:35.749232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.749043    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\") pod \"a41f28d0-2a31-4058-8bde-d08165d89340\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") "
Oct 31 04:14:35.749232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.749070    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\") pod \"a41f28d0-2a31-4058-8bde-d08165d89340\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") "
Oct 31 04:14:35.749232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.749092    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\") pod \"a41f28d0-2a31-4058-8bde-d08165d89340\" (UID: \"a41f28d0-2a31-4058-8bde-d08165d89340\") "
Oct 31 04:14:35.749467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.749445    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca" pod "a41f28d0-2a31-4058-8bde-d08165d89340" (UID: "a41f28d0-2a31-4058-8bde-d08165d89340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:35.751299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.751274    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs" pod "a41f28d0-2a31-4058-8bde-d08165d89340" (UID: "a41f28d0-2a31-4058-8bde-d08165d89340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:35.751413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.751328    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls" pod "a41f28d0-2a31-4058-8bde-d08165d89340" (UID: "a41f28d0-2a31-4058-8bde-d08165d89340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:35.751476 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.751424    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv" pod "a41f28d0-2a31-4058-8bde-d08165d89340" (UID: "a41f28d0-2a31-4058-8bde-d08165d89340"). InnerVolumeSpecName "kube-api-access-m5pgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:35.777547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.777510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.850875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.850787    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:35.850875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.850821    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/a41f28d0-2a31-4058-8bde-d08165d89340-kube-api-access-m5pgv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:35.850875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.850837    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a41f28d0-2a31-4058-8bde-d08165d89340-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:35.850875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.850849    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a41f28d0-2a31-4058-8bde-d08165d89340-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:35.965919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.965887    6931 generic.go:356] "Generic (PLEG): container finished" podID="a41f28d0-2a31-4058-8bde-d08165d89340" containerID="a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86" exitCode=0
Oct 31 04:14:35.966107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.965937    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"
Oct 31 04:14:35.966107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.965973    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" event={"ID":"a41f28d0-2a31-4058-8bde-d08165d89340","Type":"ContainerDied","Data":"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"}
Oct 31 04:14:35.966107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.966033    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f" event={"ID":"a41f28d0-2a31-4058-8bde-d08165d89340","Type":"ContainerDied","Data":"4d28b889ed6aac86ca61c2703f15bd441b02c89148ef66034c57a1695f4c5b71"}
Oct 31 04:14:35.966107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.966059    6931 scope.go:122] "RemoveContainer" containerID="a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"
Oct 31 04:14:35.975540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.975519    6931 scope.go:122] "RemoveContainer" containerID="a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"
Oct 31 04:14:35.975902 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:35.975883    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86\": container with ID starting with a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86 not found: ID does not exist" containerID="a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"
Oct 31 04:14:35.975945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:35.975911    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86"} err="failed to get container status \"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86\": rpc error: code = NotFound desc = could not find container \"a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86\": container with ID starting with a6ca81a9f587c248c5d6cd212bacb7c55f700d49261c8cda3a7c865bb1363d86 not found: ID does not exist"
Oct 31 04:14:36.018521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.018486    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:36.022594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.022560    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0d1a61d-5578597f98-q475f"]
Oct 31 04:14:36.075205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.075149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:36.121873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.121795    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41f28d0-2a31-4058-8bde-d08165d89340" path="/var/lib/kubelet/pods/a41f28d0-2a31-4058-8bde-d08165d89340/volumes"
Oct 31 04:14:36.290571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.290533    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:36.662826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:36.662790    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:37.571610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:37.571576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:37.792024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:37.791983    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:38.004768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:38.004681    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:38.210705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:38.210664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:38.429456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:38.429418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:39.027110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.027057    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:39.298534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.298481    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:39.797548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.797517    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:39.797893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.797861    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" podUID="1545d84f-898c-4831-bc2c-40fbb79153e7" containerName="maas-api" containerID="cri-o://1e0a7448218ec6374d70f06e6ac27419020cb90f33093fef53d80f8b6b6e990e" gracePeriod=30
Oct 31 04:14:39.962594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.962557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:39.989624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.989592    6931 generic.go:356] "Generic (PLEG): container finished" podID="1545d84f-898c-4831-bc2c-40fbb79153e7" containerID="1e0a7448218ec6374d70f06e6ac27419020cb90f33093fef53d80f8b6b6e990e" exitCode=0
Oct 31 04:14:39.989813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:39.989669    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" event={"ID":"1545d84f-898c-4831-bc2c-40fbb79153e7","Type":"ContainerDied","Data":"1e0a7448218ec6374d70f06e6ac27419020cb90f33093fef53d80f8b6b6e990e"}
Oct 31 04:14:40.061491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.059866    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:40.199857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.199824    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\") pod \"1545d84f-898c-4831-bc2c-40fbb79153e7\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") "
Oct 31 04:14:40.200028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.199977    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") pod \"1545d84f-898c-4831-bc2c-40fbb79153e7\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") "
Oct 31 04:14:40.200028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.199997    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\") pod \"1545d84f-898c-4831-bc2c-40fbb79153e7\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") "
Oct 31 04:14:40.200112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.200037    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\" (UniqueName: \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\") pod \"1545d84f-898c-4831-bc2c-40fbb79153e7\" (UID: \"1545d84f-898c-4831-bc2c-40fbb79153e7\") "
Oct 31 04:14:40.200197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.200143    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca" pod "1545d84f-898c-4831-bc2c-40fbb79153e7" (UID: "1545d84f-898c-4831-bc2c-40fbb79153e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:40.200409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.200389    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1545d84f-898c-4831-bc2c-40fbb79153e7-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:40.202180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.202129    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls" pod "1545d84f-898c-4831-bc2c-40fbb79153e7" (UID: "1545d84f-898c-4831-bc2c-40fbb79153e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:40.202293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.202245    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn" pod "1545d84f-898c-4831-bc2c-40fbb79153e7" (UID: "1545d84f-898c-4831-bc2c-40fbb79153e7"). InnerVolumeSpecName "kube-api-access-nfcpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:40.202293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.202259    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs" pod "1545d84f-898c-4831-bc2c-40fbb79153e7" (UID: "1545d84f-898c-4831-bc2c-40fbb79153e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:40.301758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.301713    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:40.301758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.301750    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1545d84f-898c-4831-bc2c-40fbb79153e7-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:40.301758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.301763    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcpn\" (UniqueName: \"kubernetes.io/projected/1545d84f-898c-4831-bc2c-40fbb79153e7-kube-api-access-nfcpn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:40.726119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.726079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:40.758843 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:40.758807    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:14:40.953225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.953185    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:40.996489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.996400    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz" event={"ID":"1545d84f-898c-4831-bc2c-40fbb79153e7","Type":"ContainerDied","Data":"49264c9a0b21911df73d53494053725a321864c5b250c0da5f1fff947fc10ecc"}
Oct 31 04:14:40.996489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.996467    6931 scope.go:122] "RemoveContainer" containerID="1e0a7448218ec6374d70f06e6ac27419020cb90f33093fef53d80f8b6b6e990e"
Oct 31 04:14:40.996489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:40.996410    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"
Oct 31 04:14:41.052456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:41.052416    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:41.064495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:41.064460    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4805aacc-7989d8d7bb-g9vzz"]
Oct 31 04:14:41.609795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:41.609753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:41.663558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:41.663507    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" podUID="7366b873-e392-4c6c-b5ce-b60b3daed546" containerName="payload-processing" containerID="cri-o://c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54" gracePeriod=25
Oct 31 04:14:41.914011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:41.913987    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:42.003732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.003692    6931 generic.go:356] "Generic (PLEG): container finished" podID="7366b873-e392-4c6c-b5ce-b60b3daed546" containerID="c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54" exitCode=0
Oct 31 04:14:42.003885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.003749    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"
Oct 31 04:14:42.003885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.003758    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" event={"ID":"7366b873-e392-4c6c-b5ce-b60b3daed546","Type":"ContainerDied","Data":"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"}
Oct 31 04:14:42.003885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.003812    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b" event={"ID":"7366b873-e392-4c6c-b5ce-b60b3daed546","Type":"ContainerDied","Data":"63a93be975c7cdf9742f036b410f1ac3bb26605723c9a8a52d2f3fef7c4d6c24"}
Oct 31 04:14:42.003885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.003842    6931 scope.go:122] "RemoveContainer" containerID="c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"
Oct 31 04:14:42.014284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.014256    6931 scope.go:122] "RemoveContainer" containerID="c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"
Oct 31 04:14:42.014696 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:42.014662    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54\": container with ID starting with c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54 not found: ID does not exist" containerID="c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"
Oct 31 04:14:42.014800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.014701    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54"} err="failed to get container status \"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54\": rpc error: code = NotFound desc = could not find container \"c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54\": container with ID starting with c89e7efef908b2813ae55e1f90a9ea3db5590b42c5008119e535ff8b9ae3fa54 not found: ID does not exist"
Oct 31 04:14:42.019694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.019672    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\") pod \"7366b873-e392-4c6c-b5ce-b60b3daed546\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") "
Oct 31 04:14:42.019833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.019738    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\" (UniqueName: \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\") pod \"7366b873-e392-4c6c-b5ce-b60b3daed546\" (UID: \"7366b873-e392-4c6c-b5ce-b60b3daed546\") "
Oct 31 04:14:42.020087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.020061    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume" pod "7366b873-e392-4c6c-b5ce-b60b3daed546" (UID: "7366b873-e392-4c6c-b5ce-b60b3daed546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:42.021905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.021885    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97" pod "7366b873-e392-4c6c-b5ce-b60b3daed546" (UID: "7366b873-e392-4c6c-b5ce-b60b3daed546"). InnerVolumeSpecName "kube-api-access-tqj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:42.047810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.047750    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:42.121015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.120979    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1545d84f-898c-4831-bc2c-40fbb79153e7" path="/var/lib/kubelet/pods/1545d84f-898c-4831-bc2c-40fbb79153e7/volumes"
Oct 31 04:14:42.121472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.121398    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7366b873-e392-4c6c-b5ce-b60b3daed546-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:42.121472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.121421    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqj97\" (UniqueName: \"kubernetes.io/projected/7366b873-e392-4c6c-b5ce-b60b3daed546-kube-api-access-tqj97\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:42.265985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.265901    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:42.341772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.341737    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:42.353302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.353267    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4805aacc-9f4bd74bf-tbc8b"]
Oct 31 04:14:42.486858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.486824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:42.700046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.700010    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:42.914729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:42.914636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:43.197965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:43.197859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:43.408900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:43.408867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:44.128486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.128452    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7366b873-e392-4c6c-b5ce-b60b3daed546" path="/var/lib/kubelet/pods/7366b873-e392-4c6c-b5ce-b60b3daed546/volumes"
Oct 31 04:14:44.176483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.176440    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:44.386470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.386363    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:44.661422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661341    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 04:14:44.661784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661771    6931 cpu_manager.go:414] "containerMap: removing container" podUID="ebee838b-8183-47cd-86b2-b4cee0465c56" containerName="maas-api"
Oct 31 04:14:44.661840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661785    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebee838b-8183-47cd-86b2-b4cee0465c56" containerName="maas-api"
Oct 31 04:14:44.661840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661833    6931 cpu_manager.go:414] "containerMap: removing container" podUID="a41f28d0-2a31-4058-8bde-d08165d89340" containerName="maas-api"
Oct 31 04:14:44.661840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661839    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41f28d0-2a31-4058-8bde-d08165d89340" containerName="maas-api"
Oct 31 04:14:44.661935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661850    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1545d84f-898c-4831-bc2c-40fbb79153e7" containerName="maas-api"
Oct 31 04:14:44.661935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661856    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1545d84f-898c-4831-bc2c-40fbb79153e7" containerName="maas-api"
Oct 31 04:14:44.661935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661866    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7366b873-e392-4c6c-b5ce-b60b3daed546" containerName="payload-processing"
Oct 31 04:14:44.661935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661870    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366b873-e392-4c6c-b5ce-b60b3daed546" containerName="payload-processing"
Oct 31 04:14:44.661935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661931    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366b873-e392-4c6c-b5ce-b60b3daed546" containerName="payload-processing"
Oct 31 04:14:44.662084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661941    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebee838b-8183-47cd-86b2-b4cee0465c56" containerName="maas-api"
Oct 31 04:14:44.662084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661952    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41f28d0-2a31-4058-8bde-d08165d89340" containerName="maas-api"
Oct 31 04:14:44.662084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.661960    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1545d84f-898c-4831-bc2c-40fbb79153e7" containerName="maas-api"
Oct 31 04:14:44.665388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.665369    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.691492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.691462    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-09a20a-serving-cert\""
Oct 31 04:14:44.691908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.691884    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\""
Oct 31 04:14:44.692065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.691976    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:14:44.693802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.693776    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 04:14:44.850556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.850520    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.850556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.850565    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.850784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.850750    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.850822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.850792    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th49l\" (UniqueName: \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.860954 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.860914    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:44.952438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.952338    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.952438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.952391    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th49l\" (UniqueName: \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.952438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.952426    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.952734 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.952444    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.953167 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.953133    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.954840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.954817    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.954947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.954930    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:44.977673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:44.977645    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th49l\" (UniqueName: \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\") pod \"maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:45.074011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:45.073975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:45.276875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:45.276782    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:45.579413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:45.579377    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 04:14:45.584742 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:45.584715    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5df154a_49af_4992_b83b_059b1c03399a.slice/crio-111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2 WatchSource:0}: Error finding container 111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2: Status 404 returned error can't find the container with id 111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2
Oct 31 04:14:46.028326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.028288    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" event={"ID":"c5df154a-49af-4992-b83b-059b1c03399a","Type":"ContainerStarted","Data":"8f3cc0c23b49df04f8e338c46d39b588a347614844f4c2adc16d01923a548a48"}
Oct 31 04:14:46.028481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.028355    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" event={"ID":"c5df154a-49af-4992-b83b-059b1c03399a","Type":"ContainerStarted","Data":"111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2"}
Oct 31 04:14:46.028570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.028557    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:46.077449 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.077392    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" podStartSLOduration=1.710860305 podStartE2EDuration="2.077375965s" totalImagesPullingTime="366.51566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:46.075855962 +0000 UTC m=+582.108855973" watchObservedRunningTime="2026-10-31 04:14:46.077375965 +0000 UTC m=+582.110375643"
Oct 31 04:14:46.583206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.583152    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:46.911172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:46.911123    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:47.108846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.108812    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 04:14:47.112383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.112365    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.134961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.134929    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 04:14:47.136457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.136436    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-4c2387-serving-cert\""
Oct 31 04:14:47.145727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.145694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:47.175982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.175898    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.175982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.175940    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvbw\" (UniqueName: \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.176199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.176016    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.176199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.176076    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.277304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.277260    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.277466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.277322    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvbw\" (UniqueName: \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.277466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.277377    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.277466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.277411    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.278266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.278242    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.279854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.279826    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.279940 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.279872    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.303080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.303040    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvbw\" (UniqueName: \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\") pod \"maas-api-e2e-worker-w3-4c2387-79759997df-fph2d\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.369002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.368967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:47.425603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.425564    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:47.579075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.579037    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:47.728971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:47.728890    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 04:14:47.733375 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:47.733346    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4138af25_08f1_468a_896a_bc1770415bdd.slice/crio-570bf2f2a602b23eac6e5f12ab031371e142e113a5e861be8823142c19c43496 WatchSource:0}: Error finding container 570bf2f2a602b23eac6e5f12ab031371e142e113a5e861be8823142c19c43496: Status 404 returned error can't find the container with id 570bf2f2a602b23eac6e5f12ab031371e142e113a5e861be8823142c19c43496
Oct 31 04:14:48.041633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:48.041588    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" event={"ID":"4138af25-08f1-468a-896a-bc1770415bdd","Type":"ContainerStarted","Data":"570bf2f2a602b23eac6e5f12ab031371e142e113a5e861be8823142c19c43496"}
Oct 31 04:14:48.292810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:48.292708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:48.688180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:48.688084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:48.906645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:48.906605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:49.054992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.054944    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" event={"ID":"4138af25-08f1-468a-896a-bc1770415bdd","Type":"ContainerStarted","Data":"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"}
Oct 31 04:14:49.055249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.055141    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:49.104121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.104063    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" podStartSLOduration=1.789949584 podStartE2EDuration="2.104044866s" totalImagesPullingTime="314.095282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:49.102788141 +0000 UTC m=+585.135787821" watchObservedRunningTime="2026-10-31 04:14:49.104044866 +0000 UTC m=+585.137044547"
Oct 31 04:14:49.120428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.120392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:49.331712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.331625    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:49.829445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:49.829406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:50.045715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:50.045677    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:51.448802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:51.448759    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:51.449302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:51.449040    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy" containerID="cri-o://daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3" gracePeriod=30
Oct 31 04:14:51.578172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:51.578097    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:51.981112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:51.981076    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:52.039015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:52.038984    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 04:14:52.406191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:52.406134    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:52.925667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:52.925636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:53.149919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.149856    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:53.869628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.869586    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 04:14:53.873464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.873443    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:53.888475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.888450    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-965f61-serving-cert\""
Oct 31 04:14:53.902898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.902866    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 04:14:53.936964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.936931    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:53.937354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.936980    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:53.937354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.937012    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:53.937354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:53.937089    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcqv\" (UniqueName: \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.038665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.038619    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.038845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.038681    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.038845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.038712    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.038845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.038766    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcqv\" (UniqueName: \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.039425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.039394    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.041146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.041122    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.041243 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.041148    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.061476 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.061448    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcqv\" (UniqueName: \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\") pod \"maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.063962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.063939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:54.186688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.186593    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:54.273548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.273511    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:54.488516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:54.488410    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 04:14:54.493519 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:14:54.493490    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b9045_5aea_45a8_9092_f6559243a0e9.slice/crio-7d3c6c4684a9ebe5bd1651c448e5afbc47b4730850d44c3624257d0dbe47e902 WatchSource:0}: Error finding container 7d3c6c4684a9ebe5bd1651c448e5afbc47b4730850d44c3624257d0dbe47e902: Status 404 returned error can't find the container with id 7d3c6c4684a9ebe5bd1651c448e5afbc47b4730850d44c3624257d0dbe47e902
Oct 31 04:14:55.065075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.065043    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 04:14:55.089415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.089380    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" event={"ID":"165b9045-5aea-45a8-9092-f6559243a0e9","Type":"ContainerStarted","Data":"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"}
Oct 31 04:14:55.089581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.089442    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" event={"ID":"165b9045-5aea-45a8-9092-f6559243a0e9","Type":"ContainerStarted","Data":"7d3c6c4684a9ebe5bd1651c448e5afbc47b4730850d44c3624257d0dbe47e902"}
Oct 31 04:14:55.089581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.089466    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:14:55.149598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.149543    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" podStartSLOduration=1.7175384390000001 podStartE2EDuration="2.149526565s" totalImagesPullingTime="431.988126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:14:55.147637436 +0000 UTC m=+591.180637141" watchObservedRunningTime="2026-10-31 04:14:55.149526565 +0000 UTC m=+591.182526240"
Oct 31 04:14:55.676986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:55.676944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:56.194414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.194377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:56.643130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.643098    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:56.713238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.713140    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:56.768446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768413    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768464    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768502    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768577    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768667    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768721    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768719    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:14:56.768845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768739    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:14:56.768845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768761    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768788    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.768845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.768828    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\") pod \"9c8d4620-d98c-41da-979f-539f7846df0e\" (UID: \"9c8d4620-d98c-41da-979f-539f7846df0e\") "
Oct 31 04:14:56.769183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.769118    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.769183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.769138    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.769446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.769258    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:14:56.769446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.769375    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:14:56.769638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.769613    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:56.771334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.771307    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:56.771469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.771379    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:14:56.771632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.771609    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "kube-api-access-xsglb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:56.782288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.779626    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy" pod "9c8d4620-d98c-41da-979f-539f7846df0e" (UID: "9c8d4620-d98c-41da-979f-539f7846df0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:14:56.869839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869805    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsglb\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-kube-api-access-xsglb\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.869839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869836    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.869839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869845    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8d4620-d98c-41da-979f-539f7846df0e-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.870091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869853    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.870091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869863    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8d4620-d98c-41da-979f-539f7846df0e-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.870091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869871    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8d4620-d98c-41da-979f-539f7846df0e-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.870091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.869878    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8d4620-d98c-41da-979f-539f7846df0e-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:14:56.881270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:56.881236    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:57.101457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.101421    6931 generic.go:356] "Generic (PLEG): container finished" podID="9c8d4620-d98c-41da-979f-539f7846df0e" containerID="daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3" exitCode=0
Oct 31 04:14:57.101634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.101474    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"
Oct 31 04:14:57.101634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.101502    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" event={"ID":"9c8d4620-d98c-41da-979f-539f7846df0e","Type":"ContainerDied","Data":"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"}
Oct 31 04:14:57.101634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.101558    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92" event={"ID":"9c8d4620-d98c-41da-979f-539f7846df0e","Type":"ContainerDied","Data":"a871edc19376c59e05751fe562e116be688b642524753d13cb7b40c139e88a9a"}
Oct 31 04:14:57.101634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.101581    6931 scope.go:122] "RemoveContainer" containerID="daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"
Oct 31 04:14:57.112011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.111989    6931 scope.go:122] "RemoveContainer" containerID="daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"
Oct 31 04:14:57.112483 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:14:57.112461    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3\": container with ID starting with daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3 not found: ID does not exist" containerID="daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"
Oct 31 04:14:57.112571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.112492    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3"} err="failed to get container status \"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3\": rpc error: code = NotFound desc = could not find container \"daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3\": container with ID starting with daf786c42fa3c81b04d8e4f4a993801663ac6039738bc0cc5f8a512d25cecea3 not found: ID does not exist"
Oct 31 04:14:57.182829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.182794    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:57.184114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.184073    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:57.187094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:57.187071    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44f032ad-openshift-default-5f845cc957-x7d92"]
Oct 31 04:14:58.121226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:58.121185    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" path="/var/lib/kubelet/pods/9c8d4620-d98c-41da-979f-539f7846df0e/volumes"
Oct 31 04:14:58.291381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:58.291349    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:59.002404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:59.002360    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:59.621787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:59.621750    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:59.852971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:14:59.852936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:00.001492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.001417    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 04:15:00.001851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.001839    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy"
Oct 31 04:15:00.001904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.001852    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy"
Oct 31 04:15:00.001963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.001954    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8d4620-d98c-41da-979f-539f7846df0e" containerName="istio-proxy"
Oct 31 04:15:00.006755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.006736    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.025042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.025010    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-439989-serving-cert\""
Oct 31 04:15:00.039776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.039739    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 04:15:00.087652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.087614    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:00.100086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.100051    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.100241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.100097    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6s8h\" (UniqueName: \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.100241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.100203    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.100341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.100297    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.201595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.201562    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.201843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.201820    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.201923 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:15:00.201909    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-serving-cert: secret "maas-api-e2e-worker-w1-439989-serving-cert" not found
Oct 31 04:15:00.201973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.201957    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s8h\" (UniqueName: \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.202013 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:15:00.201989    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls podName:8c3e0aa9-2709-41ea-b013-d27f958fe7db nodeName:}" failed. No retries permitted until 2026-10-31 04:15:00.70196657 +0000 UTC m=+596.734966240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls") pod "maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" (UID: "8c3e0aa9-2709-41ea-b013-d27f958fe7db") : secret "maas-api-e2e-worker-w1-439989-serving-cert" not found
Oct 31 04:15:00.202063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.202047    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.202671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.202652    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.204245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.204218    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.229124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.229088    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s8h\" (UniqueName: \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.296442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.296404    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:00.501973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.501935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:00.707774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.707676    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.710035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.710004    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") pod \"maas-api-e2e-worker-w1-439989-6f8d444988-58qz4\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:00.919608 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:00.919554    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:01.101957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:01.101928    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 04:15:01.228829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:01.228745    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 04:15:01.233200 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:15:01.233148    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3e0aa9_2709_41ea_b013_d27f958fe7db.slice/crio-c82f94cced1825bd4a7f7ba01bc62bb86f7671af93f76d1f0702caa9e6a2001e WatchSource:0}: Error finding container c82f94cced1825bd4a7f7ba01bc62bb86f7671af93f76d1f0702caa9e6a2001e: Status 404 returned error can't find the container with id c82f94cced1825bd4a7f7ba01bc62bb86f7671af93f76d1f0702caa9e6a2001e
Oct 31 04:15:01.997641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:01.997526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:02.129046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.129005    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" event={"ID":"8c3e0aa9-2709-41ea-b013-d27f958fe7db","Type":"ContainerStarted","Data":"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"}
Oct 31 04:15:02.129263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.129069    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" event={"ID":"8c3e0aa9-2709-41ea-b013-d27f958fe7db","Type":"ContainerStarted","Data":"c82f94cced1825bd4a7f7ba01bc62bb86f7671af93f76d1f0702caa9e6a2001e"}
Oct 31 04:15:02.129263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.129200    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:02.171689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.171633    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" podStartSLOduration=2.738956553 podStartE2EDuration="3.171608136s" totalImagesPullingTime="432.651583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.170933995 +0000 UTC m=+598.203933697" watchObservedRunningTime="2026-10-31 04:15:02.171608136 +0000 UTC m=+598.204607815"
Oct 31 04:15:02.220812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.220773    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:02.442370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.442323    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:02.656228 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.656185    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:02.891623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:02.891583    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:04.425264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:04.425230    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:06.402843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:06.402802    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:06.673445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:06.673343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:08.142837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:08.142810    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 04:15:08.658315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:08.658282    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:10.619179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:10.619133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:10.760946 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:15:10.760902    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:15:11.870947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:11.870907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:12.494880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:12.494817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:12.714188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:12.714127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:12.963719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:12.963682    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:14.984363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:14.984327    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:16.777257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:16.777224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:17.265004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:17.264965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:18.079016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:18.078981    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:18.696946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:18.696910    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:18.925087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:18.925042    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:19.378326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:19.378284    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:21.269234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:21.269194    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:22.030978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:22.030942    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:22.727418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:22.727370    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:22.959809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:22.959774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:23.573527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:23.573483    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:25.550733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:25.550690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:27.466572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:27.466535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:28.255572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:28.255529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:29.302239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:29.302178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:29.718944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:29.718855    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:31.111242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:31.111202    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:33.003576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:33.003527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:33.712278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:33.712239    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:34.804553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:34.804516    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:36.690025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:36.689986    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:38.502988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:38.502943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:40.402197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:40.402144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:40.758646 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:15:40.758568    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:15:42.189752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:42.189716    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:44.018012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:44.017967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:45.822694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:45.822653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:46.650021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:46.649978    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:47.713801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:47.713762    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:49.507633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:49.507591    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:51.409581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:51.409544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:53.329276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:53.329237    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:55.339770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.339734    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:55.445714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.445676    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 04:15:55.450105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.450078    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.478920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.478877    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"openshift-service-ca.crt\""
Oct 31 04:15:55.478920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.478877    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"premium-w0-965f61-kserve-self-signed-certs\""
Oct 31 04:15:55.479133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.478931    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"kube-root-ca.crt\""
Oct 31 04:15:55.479133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.479027    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"default-dockercfg-gvlvp\""
Oct 31 04:15:55.480776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.480757    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 04:15:55.574969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.574933    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.575152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.574990    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.575152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.575080    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.575284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.575229    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx225\" (UniqueName: \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.575335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.575291    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.575376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.575350    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.601213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.601095    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:55.676310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676267    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676355    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676385    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676451    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx225\" (UniqueName: \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676478    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676533    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676715    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676844 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676802    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.676943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.676922    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.678810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.678782    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.679169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.679123    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.699382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.699351    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx225\" (UniqueName: \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\") pod \"premium-w0-965f61-kserve-687bd6b55c-wc7kg\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") " pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.763374 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.763340    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:55.859865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:55.859776    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:56.064180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:56.064122    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 04:15:56.068100 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:15:56.068074    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb94979e4_1d5a_473e_9e91_75af7a823ea5.slice/crio-30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2 WatchSource:0}: Error finding container 30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2: Status 404 returned error can't find the container with id 30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2
Oct 31 04:15:56.427180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:56.427119    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" event={"ID":"b94979e4-1d5a-473e-9e91-75af7a823ea5","Type":"ContainerStarted","Data":"30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2"}
Oct 31 04:15:57.312385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:57.312352    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:57.434110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:57.434069    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" event={"ID":"b94979e4-1d5a-473e-9e91-75af7a823ea5","Type":"ContainerStarted","Data":"97ac8c4d17d3de2a4f1c2e4ad0ce993fef86491e55e07783b527ac27dfe4a25a"}
Oct 31 04:15:57.475795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:57.475728    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podStartSLOduration=1.287059199 podStartE2EDuration="2.475710596s" totalImagesPullingTime="1.188651397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:57.474499291 +0000 UTC m=+653.507498989" watchObservedRunningTime="2026-10-31 04:15:57.475710596 +0000 UTC m=+653.508710276"
Oct 31 04:15:57.764560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:57.764520    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:15:59.198220 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:15:59.198190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:00.901343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:00.901297    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:01.119988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:01.119953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:02.906010 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:02.905974    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:04.708964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:04.708930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:05.763605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:05.763563    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:16:05.776060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:05.776032    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:16:06.496237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:06.496206    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:16:06.508514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:06.508479    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:06.908434 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:06.908390    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:08.409271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:08.409234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:10.307055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:10.307015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:10.759637 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:16:10.759550    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:16:12.109313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:12.109277    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:13.914392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:13.914360    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:15.705242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:15.705204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:17.610914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:17.610783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:19.373939 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:19.373903    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:21.315975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:21.315938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:23.276423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:23.276382    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:25.129628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:25.129584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:26.914699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:26.914664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:28.809350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:28.809315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:30.826887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:30.826848    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:32.809749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:32.809712    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:34.802557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:34.802513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:36.607391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:36.607313    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:38.401632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:38.401590    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:40.226992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:40.226950    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:40.758771 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:16:40.758736    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:16:42.098074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:42.098039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:43.917178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:43.917119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:45.713987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:45.713943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:47.590679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:47.590636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:49.417042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:49.417001    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:50.601735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.601693    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:16:50.606906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.606882    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.621326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.621296    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-09a20a-openshift-default-dockercfg-hf9qb\""
Oct 31 04:16:50.633517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.633486    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:16:50.717275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717243    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717277    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717306    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717356    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpfh\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717441    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717464    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717489    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717628    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.717749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.717737    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.818918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.818883    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.818930    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.818970    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.818999    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819014    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819197    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819229    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpfh\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819301    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819349    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819391    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819458    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819635    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819780    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.819859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.819812    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.821314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.821287    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.821824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.821805    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.840249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.840223    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.840421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.840404    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpfh\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:50.919457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:50.919369    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:51.180776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.180678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:51.232595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.232556    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:16:51.237233 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:16:51.237206    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f51be56_c3fd_4fe1_a020_cd45da9043d5.slice/crio-182f03f969320674fb6d01d50f5316394cce1838b654333ae634b398d117b2f1 WatchSource:0}: Error finding container 182f03f969320674fb6d01d50f5316394cce1838b654333ae634b398d117b2f1: Status 404 returned error can't find the container with id 182f03f969320674fb6d01d50f5316394cce1838b654333ae634b398d117b2f1
Oct 31 04:16:51.239704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.239670    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:16:51.239772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.239733    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:16:51.239772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.239760    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:16:51.712150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.712113    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" event={"ID":"3f51be56-c3fd-4fe1-a020-cd45da9043d5","Type":"ContainerStarted","Data":"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"}
Oct 31 04:16:51.712827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.712201    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" event={"ID":"3f51be56-c3fd-4fe1-a020-cd45da9043d5","Type":"ContainerStarted","Data":"182f03f969320674fb6d01d50f5316394cce1838b654333ae634b398d117b2f1"}
Oct 31 04:16:51.754321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.754262    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podStartSLOduration=1.7542470300000002 podStartE2EDuration="1.75424703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:51.752325018 +0000 UTC m=+707.785324722" watchObservedRunningTime="2026-10-31 04:16:51.75424703 +0000 UTC m=+707.787246781"
Oct 31 04:16:51.920412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:51.920373    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:52.921915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:52.921879    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 04:16:52.922357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:52.921931    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 04:16:53.004589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:53.004555    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:53.920232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:53.920190    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 04:16:53.920421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:53.920250    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 04:16:54.809937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:54.809895    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:54.920530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:54.920495    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 04:16:54.920711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:54.920547    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 04:16:55.920140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:55.920101    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 04:16:55.920567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:55.920188    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 04:16:55.973672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:55.973641    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:56.606141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:56.606101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:56.920816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:56.920732    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 04:16:56.920816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:56.920790    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 04:16:57.545209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:57.545149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:57.944457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:57.944302    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:57.945509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:57.945490    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:57.946259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:57.946223    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:16:58.512279 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:16:58.512244    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:00.255282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:00.255243    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:02.102335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:02.102302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:03.630271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:03.630222    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:03.861513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:03.861480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:04.073174 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:04.073123    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:04.733727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:04.733693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:05.989587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:05.989527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:07.819474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:07.819414    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:09.604339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:09.604292    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:10.763718 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:17:10.763679    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:17:11.414927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:11.414885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:13.405126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:13.405082    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:15.291069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:15.290974    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:17.089829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:17.089777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:18.904035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:18.903991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:20.813695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:20.813651    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:22.768789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:22.768753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:23.108173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:23.108117    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:24.374112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.374073    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:17:24.378012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.377990    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.406389 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.406351    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:17:24.481462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481427    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481561    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481595    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481650    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481753    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481806    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn74p\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481880    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481904    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.481952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.481942    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583017    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583061    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583093    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583119    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583147    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn74p\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583185    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583213    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583257    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583291    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583501    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583663    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.583874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583848    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.584002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.583959    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.584122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.584003    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.585679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.585653    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.585809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.585790    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.606788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.606759    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.606950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.606925    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn74p\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gnfkb\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:24.696086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:24.695997    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:25.003877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.003765    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:25.021609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.021571    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:17:25.026394 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:17:25.026366    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3ad619_d9db_4a40_8a33_04f861625790.slice/crio-207b7cdc14c9cc3693ddd69b63ea96a48bf93021182f1ee499280889fb123a06 WatchSource:0}: Error finding container 207b7cdc14c9cc3693ddd69b63ea96a48bf93021182f1ee499280889fb123a06: Status 404 returned error can't find the container with id 207b7cdc14c9cc3693ddd69b63ea96a48bf93021182f1ee499280889fb123a06
Oct 31 04:17:25.028651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.028619    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:17:25.028735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.028684    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:17:25.028735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.028714    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:17:25.915526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.915485    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" event={"ID":"9d3ad619-d9db-4a40-8a33-04f861625790","Type":"ContainerStarted","Data":"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"}
Oct 31 04:17:25.915971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.915554    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" event={"ID":"9d3ad619-d9db-4a40-8a33-04f861625790","Type":"ContainerStarted","Data":"207b7cdc14c9cc3693ddd69b63ea96a48bf93021182f1ee499280889fb123a06"}
Oct 31 04:17:25.995562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:25.995499    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podStartSLOduration=1.995483726 podStartE2EDuration="1.995483726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:25.982487426 +0000 UTC m=+742.015487117" watchObservedRunningTime="2026-10-31 04:17:25.995483726 +0000 UTC m=+742.028483405"
Oct 31 04:17:26.696294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:26.696256    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:26.697847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:26.697819    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:26.697949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:26.697870    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:26.989352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:26.989259    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:27.696800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:27.696769    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:27.696969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:27.696833    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:28.697304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:28.697262    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:28.697702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:28.697328    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:29.076030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:29.075991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:29.696558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:29.696525    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:29.696735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:29.696575    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:30.697093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:30.697058    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:30.697517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:30.697112    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:30.939924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:30.939874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:31.696529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:31.696494    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:17:31.696735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:31.696552    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 04:17:32.748083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:32.748032    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:33.747180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:33.746294    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:17:33.747180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:33.746373    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:17:33.821605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:33.821570    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:33.822428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:33.822407    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:33.823371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:33.823349    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:17:34.842181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:34.839418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:36.726621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:36.726579    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:38.626916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.626878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:38.647923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.647882    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 04:17:38.652128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.652100    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.668620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.668589    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"distinct-w1-439989-kserve-self-signed-certs\""
Oct 31 04:17:38.680710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.680678    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"kube-root-ca.crt\""
Oct 31 04:17:38.680710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.680678    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"default-dockercfg-bxjc6\""
Oct 31 04:17:38.680710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.680678    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"openshift-service-ca.crt\""
Oct 31 04:17:38.685304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.685281    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 04:17:38.755612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.755575    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.755782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.755637    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgnn\" (UniqueName: \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.755824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.755764    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.755864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.755822    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.755983 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.755962    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.756019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.756006    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.839455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.839418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:38.856648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856606    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.856648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856651    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.856914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856703    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.856914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856774    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgnn\" (UniqueName: \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.856914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856832    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.856914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.856873    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.857270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.857244    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.857511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.857488    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.857602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.857586    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.859673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.859644    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.860038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.859991    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.882388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.882302    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgnn\" (UniqueName: \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\") pod \"distinct-w1-439989-kserve-79c5d77d68-bn2br\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") " pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:38.966246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:38.966148    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:39.267665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:39.267569    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 04:17:39.272329 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:17:39.272301    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9403b8_e1b8_4f52_ace8_0bf1714353df.slice/crio-cf5fcacf3f13caca3277d7a2063dd9e83f0ad926248371ea5b310881bcd96311 WatchSource:0}: Error finding container cf5fcacf3f13caca3277d7a2063dd9e83f0ad926248371ea5b310881bcd96311: Status 404 returned error can't find the container with id cf5fcacf3f13caca3277d7a2063dd9e83f0ad926248371ea5b310881bcd96311
Oct 31 04:17:40.032478 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:40.032446    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" event={"ID":"8f9403b8-e1b8-4f52-ace8-0bf1714353df","Type":"ContainerStarted","Data":"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"}
Oct 31 04:17:40.032870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:40.032490    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" event={"ID":"8f9403b8-e1b8-4f52-ace8-0bf1714353df","Type":"ContainerStarted","Data":"cf5fcacf3f13caca3277d7a2063dd9e83f0ad926248371ea5b310881bcd96311"}
Oct 31 04:17:40.073560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:40.073507    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" podStartSLOduration=2.073493732 podStartE2EDuration="2.073493732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:40.073334257 +0000 UTC m=+756.106333935" watchObservedRunningTime="2026-10-31 04:17:40.073493732 +0000 UTC m=+756.106493411"
Oct 31 04:17:40.555178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:40.555131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:40.758400 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:17:40.758355    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:17:40.967020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:40.966946    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:42.464394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:42.464356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:44.473720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:44.473687    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:46.295446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:46.295410    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:48.192711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:48.192672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:48.967107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:48.967070    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:48.980055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:48.980029    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:49.096854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:49.096822    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:17:50.223456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:50.223415    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:51.190177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:51.189283    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:17:51.190177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:51.189368    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:17:52.321071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.321034    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:52.474363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.474327    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:17:52.479182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.479140    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.507788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.507753    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"openshift-service-ca.crt\""
Oct 31 04:17:52.507788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.507752    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"kube-root-ca.crt\""
Oct 31 04:17:52.507788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.507793    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"default-dockercfg-qcqb8\""
Oct 31 04:17:52.508092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.507859    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-09a20a\"/\"e2e-conflict-b-ba60a444-kserve-self-signed-certs\""
Oct 31 04:17:52.510808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.510764    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:17:52.526622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526582    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.526622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526622    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.526809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526641    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.526809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526786    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.526881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526819    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.526939 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.526919    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfjr\" (UniqueName: \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.627956    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628000    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628068    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfjr\" (UniqueName: \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628191    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628213    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628244    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628467    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628561    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.628659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.628600    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.630483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.630447    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:52.630891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.630867    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.630993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.630894    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.651083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.651052    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfjr\" (UniqueName: \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\") pod \"e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") " pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:52.802853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:52.802815    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:53.106423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:53.106382    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:17:53.110366 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:17:53.110338    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3925f2ea_adb0_434d_82d6_2aa94dc3a18e.slice/crio-527aaa9bcef53c0abad28f5195be438258cad3a163f61e17f35ae7ca4098e92f WatchSource:0}: Error finding container 527aaa9bcef53c0abad28f5195be438258cad3a163f61e17f35ae7ca4098e92f: Status 404 returned error can't find the container with id 527aaa9bcef53c0abad28f5195be438258cad3a163f61e17f35ae7ca4098e92f
Oct 31 04:17:53.122200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:53.122172    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" event={"ID":"3925f2ea-adb0-434d-82d6-2aa94dc3a18e","Type":"ContainerStarted","Data":"527aaa9bcef53c0abad28f5195be438258cad3a163f61e17f35ae7ca4098e92f"}
Oct 31 04:17:54.105580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:54.105540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:54.127986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:54.127947    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" event={"ID":"3925f2ea-adb0-434d-82d6-2aa94dc3a18e","Type":"ContainerStarted","Data":"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"}
Oct 31 04:17:54.169227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:54.169165    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" podStartSLOduration=2.169134243 podStartE2EDuration="2.169134243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:54.167569171 +0000 UTC m=+770.200568860" watchObservedRunningTime="2026-10-31 04:17:54.169134243 +0000 UTC m=+770.202133923"
Oct 31 04:17:54.803727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:54.803687    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:17:56.008307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:56.008264    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:56.223706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:56.223664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:57.813281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:57.813240    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:59.600440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:17:59.600405    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:01.511959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:01.511924    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:02.803271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:02.803231    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:18:02.815752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:02.815724    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:18:03.188565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:03.188470    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:18:03.326691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:03.326653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:05.050364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:05.050330    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:05.648645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:05.648605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:05.871494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:05.871460    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:07.025850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.025800    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:07.088503 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.088470    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 04:18:07.094119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.094085    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.123398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.123368    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 04:18:07.126772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.126739    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-965f61\"/\"distinct-w0-965f61-kserve-self-signed-certs\""
Oct 31 04:18:07.180917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.180878    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.181107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.180949    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92t4b\" (UniqueName: \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.181204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.181102    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.181204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.181147    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.181331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.181308    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.181401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.181385    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.257842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.257802    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:07.282723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282681    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.282723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282723    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.282970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282782    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.282970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282842    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.282970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282870    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.282970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.282930    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92t4b\" (UniqueName: \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.283396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.283346    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.283535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.283509    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.283700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.283599    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.285350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.285303    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.285869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.285851    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.306720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.306653    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92t4b\" (UniqueName: \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\") pod \"distinct-w0-965f61-kserve-696d7476-84n6h\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") " pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.407598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.407562    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:07.711023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:07.710947    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 04:18:07.714184 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:18:07.714139    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb877bbfa_18c7_4a63_90db_81fd73c8c02e.slice/crio-cc8347bad7186e76fe469c49fc7e99bd47b38930bd4b00402b11cbd2b6fd825d WatchSource:0}: Error finding container cc8347bad7186e76fe469c49fc7e99bd47b38930bd4b00402b11cbd2b6fd825d: Status 404 returned error can't find the container with id cc8347bad7186e76fe469c49fc7e99bd47b38930bd4b00402b11cbd2b6fd825d
Oct 31 04:18:08.203800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:08.203762    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" event={"ID":"b877bbfa-18c7-4a63-90db-81fd73c8c02e","Type":"ContainerStarted","Data":"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"}
Oct 31 04:18:08.203800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:08.203806    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" event={"ID":"b877bbfa-18c7-4a63-90db-81fd73c8c02e","Type":"ContainerStarted","Data":"cc8347bad7186e76fe469c49fc7e99bd47b38930bd4b00402b11cbd2b6fd825d"}
Oct 31 04:18:08.245531 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:08.245473    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podStartSLOduration=1.245455842 podStartE2EDuration="1.245455842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:08.245230298 +0000 UTC m=+784.278229977" watchObservedRunningTime="2026-10-31 04:18:08.245455842 +0000 UTC m=+784.278455526"
Oct 31 04:18:08.408379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:08.408341    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:08.922716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:08.922666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:10.759026 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:18:10.758989    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:18:10.788713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:10.788672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:12.613990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:12.613943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:13.887882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:13.887846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:14.894927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:14.894890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:16.709947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.709913    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 04:18:16.714253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.714224    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726188    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726235    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726313    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726357    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726399    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskkt\" (UniqueName: \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.726606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.726467    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.732624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.732598    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 04:18:16.741980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.741956    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-439989\"/\"distinct-2-w1-439989-kserve-self-signed-certs\""
Oct 31 04:18:16.827612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827569    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.827816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827640    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskkt\" (UniqueName: \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.827816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827709    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.827816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827809    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.827980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827855    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.827980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.827887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.828720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.828435    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.828720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.828683    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.829054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.829029    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.830943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.830916    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.831186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.831138    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.851427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.851390    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskkt\" (UniqueName: \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\") pod \"distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") " pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:16.869249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:16.869212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:17.027756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:17.027674    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:17.223180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:17.223135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:17.336101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:17.336059    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 04:18:17.340864 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:18:17.340839    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae8f375_9d23_4b41_ae0f_3d4cd4ae4d12.slice/crio-8d9adcea5b90a00d72565b806ec95a22de904d55b1540c8e65728bc88e4b4b89 WatchSource:0}: Error finding container 8d9adcea5b90a00d72565b806ec95a22de904d55b1540c8e65728bc88e4b4b89: Status 404 returned error can't find the container with id 8d9adcea5b90a00d72565b806ec95a22de904d55b1540c8e65728bc88e4b4b89
Oct 31 04:18:17.408531 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:17.408499    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:17.422467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:17.422435    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:18.331783 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:18.331744    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" event={"ID":"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12","Type":"ContainerStarted","Data":"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"}
Oct 31 04:18:18.332199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:18.331794    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" event={"ID":"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12","Type":"ContainerStarted","Data":"8d9adcea5b90a00d72565b806ec95a22de904d55b1540c8e65728bc88e4b4b89"}
Oct 31 04:18:18.343104 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:18.343075    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:18:18.372468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:18.372415    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" podStartSLOduration=2.372403958 podStartE2EDuration="2.372403958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:18.3717695 +0000 UTC m=+794.404769179" watchObservedRunningTime="2026-10-31 04:18:18.372403958 +0000 UTC m=+794.405403683"
Oct 31 04:18:19.027974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.027941    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:19.087517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.087483    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:19.312641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.312602    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:18:19.315928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.315907    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.331934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.331906    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-439989-openshift-default-dockercfg-9247r\""
Oct 31 04:18:19.346620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.346585    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:18:19.352964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.352936    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.352978    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353072    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353228    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353387    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353425    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353604    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353689    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqz5\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.353767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.353731    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455346    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455390    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455424    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455451    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqz5\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455474    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455523    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455552    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455584    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455645    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.455928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455832    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.456025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.455928    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.456025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.456004    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.456302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.456283    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.456388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.456371    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.457855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.457833    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.458081 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.458061    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.479688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.479663    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.479921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.479902    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqz5\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.628911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.628816    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:19.660340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.660307    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:18:19.799033 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.798985    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:19.998331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:19.997084    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:18:20.007092 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:18:20.007061    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a35242b_41d6_4821_912e_43b4ed6bb53f.slice/crio-5571adaefc54b55a35764085c3ff9425a67c1692b86f108ca2dee8978f9cee61 WatchSource:0}: Error finding container 5571adaefc54b55a35764085c3ff9425a67c1692b86f108ca2dee8978f9cee61: Status 404 returned error can't find the container with id 5571adaefc54b55a35764085c3ff9425a67c1692b86f108ca2dee8978f9cee61
Oct 31 04:18:20.009550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.009513    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:20.009636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.009588    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:20.009636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.009617    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:20.344797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.344742    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" event={"ID":"1a35242b-41d6-4821-912e-43b4ed6bb53f","Type":"ContainerStarted","Data":"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"}
Oct 31 04:18:20.345309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.344812    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" event={"ID":"1a35242b-41d6-4821-912e-43b4ed6bb53f","Type":"ContainerStarted","Data":"5571adaefc54b55a35764085c3ff9425a67c1692b86f108ca2dee8978f9cee61"}
Oct 31 04:18:20.414323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.414266    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podStartSLOduration=1.414252715 podStartE2EDuration="1.414252715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:20.401017701 +0000 UTC m=+796.434017399" watchObservedRunningTime="2026-10-31 04:18:20.414252715 +0000 UTC m=+796.447252388"
Oct 31 04:18:20.630438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.630326    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:20.860717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.860624    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:18:20.864747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.864722    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.892786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.892708    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978720    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978782    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978809    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978827    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978844    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfs8\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978870    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978885    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978909    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.979103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.978967    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:20.994897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:20.994864    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:21.079624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079592    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079635    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079682    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079707    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079724    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079742    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfs8\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079771    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.079812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079794    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.079821    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.080037    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.080057    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.080329    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.080480    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.080632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.080614    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.082126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.082103    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.082393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.082375    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.102540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.102509    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfs8\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.103008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.102982    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.178383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.178288    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:21.522182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.522053    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:18:21.525735 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:18:21.525709    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45588b4_b059_428e_977a_d432a1f27403.slice/crio-a7ba200265c77d9a048a58e47a600bebd9e8d3416e22135cb8051b170c044e9c WatchSource:0}: Error finding container a7ba200265c77d9a048a58e47a600bebd9e8d3416e22135cb8051b170c044e9c: Status 404 returned error can't find the container with id a7ba200265c77d9a048a58e47a600bebd9e8d3416e22135cb8051b170c044e9c
Oct 31 04:18:21.549451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.541483    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:21.549451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.541575    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:21.549451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.541618    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:18:21.650181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.648283    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:21.650181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:21.648344    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:22.380184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.380004    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" event={"ID":"b45588b4-b059-428e-977a-d432a1f27403","Type":"ContainerStarted","Data":"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"}
Oct 31 04:18:22.380184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.380065    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" event={"ID":"b45588b4-b059-428e-977a-d432a1f27403","Type":"ContainerStarted","Data":"a7ba200265c77d9a048a58e47a600bebd9e8d3416e22135cb8051b170c044e9c"}
Oct 31 04:18:22.475186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.470640    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podStartSLOduration=2.470618204 podStartE2EDuration="2.470618204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:22.46448664 +0000 UTC m=+798.497486348" watchObservedRunningTime="2026-10-31 04:18:22.470618204 +0000 UTC m=+798.503617884"
Oct 31 04:18:22.630666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.630573    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:22.631235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.631203    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:22.825878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:22.825838    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:23.178487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:23.178453    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:23.179740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:23.179711    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.179883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:23.179759    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:23.634184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:23.630577    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.634184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:23.630637    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:24.181187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:24.179782    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.181187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:24.179848    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:24.619821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:24.619780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:24.650182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:24.638610    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.650182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:24.638686    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:25.183180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:25.182780    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.183180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:25.182845    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:25.631492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:25.631454    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.631811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:25.631772    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:26.181883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:26.181468    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:26.181883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:26.181527    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:26.523255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:26.523166    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:26.631188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:26.630533    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 04:18:26.631188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:26.630591    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 04:18:27.033187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.032499    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:27.070275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.070244    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:27.179318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.179275    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:27.179492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.179336    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:27.426147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.426064    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 04:18:27.634457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.634425    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:27.634723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.634705    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:27.635342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:27.635325    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:18:28.178682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:28.178642    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 04:18:28.178903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:28.178715    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 04:18:28.632409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:28.627648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:30.234183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:30.224289    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:18:30.234183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:30.224360    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:18:30.765032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:30.764995    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:30.767270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:30.767223    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:30.771177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:30.769353    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:18:31.512179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:31.509429    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:32.636992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:32.636955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:33.459941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:33.459906    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:34.689857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:34.689813    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" podUID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" containerName="main" containerID="cri-o://c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2" gracePeriod=45
Oct 31 04:18:34.953059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:34.952998    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:18:35.040145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040112    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040177    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040196    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040225    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\" (UniqueName: \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040325    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040343    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\") pod \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\" (UID: \"3925f2ea-adb0-434d-82d6-2aa94dc3a18e\") "
Oct 31 04:18:35.040634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040429    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:35.040634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040526    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:35.040713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040647    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:35.040804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040790    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.040836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040807    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.040836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.040820    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.042515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.042490    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:18:35.042762 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.042741    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:18:35.042836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.042774    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr" pod "3925f2ea-adb0-434d-82d6-2aa94dc3a18e" (UID: "3925f2ea-adb0-434d-82d6-2aa94dc3a18e"). InnerVolumeSpecName "kube-api-access-xrfjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:18:35.141546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.141507    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.141546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.141535    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfjr\" (UniqueName: \"kubernetes.io/projected/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-kube-api-access-xrfjr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.141546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.141545    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3925f2ea-adb0-434d-82d6-2aa94dc3a18e-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:18:35.307409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.307369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:35.485541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.485508    6931 generic.go:356] "Generic (PLEG): container finished" podID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" containerID="c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2" exitCode=0
Oct 31 04:18:35.485695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.485560    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"
Oct 31 04:18:35.485695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.485590    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" event={"ID":"3925f2ea-adb0-434d-82d6-2aa94dc3a18e","Type":"ContainerDied","Data":"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"}
Oct 31 04:18:35.485695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.485643    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr" event={"ID":"3925f2ea-adb0-434d-82d6-2aa94dc3a18e","Type":"ContainerDied","Data":"527aaa9bcef53c0abad28f5195be438258cad3a163f61e17f35ae7ca4098e92f"}
Oct 31 04:18:35.485695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.485663    6931 scope.go:122] "RemoveContainer" containerID="c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"
Oct 31 04:18:35.496496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.496476    6931 scope.go:122] "RemoveContainer" containerID="c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"
Oct 31 04:18:35.496825 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:18:35.496802    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2\": container with ID starting with c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2 not found: ID does not exist" containerID="c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"
Oct 31 04:18:35.496882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.496836    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2"} err="failed to get container status \"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2\": rpc error: code = NotFound desc = could not find container \"c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2\": container with ID starting with c770134b7f21685abe7af38f0baaaaf5d12c57b143d0386dac8a32a2499eabc2 not found: ID does not exist"
Oct 31 04:18:35.536504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.536470    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:18:35.547413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:35.547385    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-09a20a/e2e-conflict-b-ba60a444-kserve-648c5dbdb7-pgwcr"]
Oct 31 04:18:36.125457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:36.125427    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" path="/var/lib/kubelet/pods/3925f2ea-adb0-434d-82d6-2aa94dc3a18e/volumes"
Oct 31 04:18:37.108770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:37.108734    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:38.842762 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:38.842727    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:39.057727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:39.057689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:40.761840 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:18:40.761798    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:18:40.788121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:40.788088    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:41.795907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:41.795859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:42.691636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:42.691604    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:44.456028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:44.455938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:44.677251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:44.677215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:44.893050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:44.893006    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:46.704801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:46.704766    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:48.212056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:48.212022    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:49.521322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:49.521287    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:50.525635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:50.525598    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:52.344080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:52.344038    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:54.220261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:54.220122    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:54.639221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:54.639178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:56.185650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:56.185616    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:56.404299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:56.404256    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:57.289247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:57.289206    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:57.515947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:57.515907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:58.675115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:18:58.675072    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:00.614628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:00.614544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:02.510132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:02.510096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:04.409822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:04.409778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:06.210016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:06.209982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:08.010955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.010914    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:08.455593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.455557    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:19:08.456353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.456327    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" containerName="main"
Oct 31 04:19:08.456353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.456346    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" containerName="main"
Oct 31 04:19:08.456507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.456448    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3925f2ea-adb0-434d-82d6-2aa94dc3a18e" containerName="main"
Oct 31 04:19:08.461098 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.461081    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.475352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.475330    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c4006d-openshift-default-dockercfg-h2s7k\""
Oct 31 04:19:08.489392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.489356    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:19:08.510114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510079    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510143    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510292    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510322    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510341    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510490    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510569    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9slc\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510606    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.510645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.510635    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611411    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611468    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611498    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611539    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611591    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9slc\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611618    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611641    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611674    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.611906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611752    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.612067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611908    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.612067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611943    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.612067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.611978    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.612242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.612225    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.612452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.612430    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.613977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.613952    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.614358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.614337    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.634517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.634489    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.634636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.634570    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9slc\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:08.773688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:08.773597    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:09.096378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.096343    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:19:09.100408 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:19:09.100383    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41691951_56fe_4f9a_92ac_2c45175692d3.slice/crio-8f3e2a751d55f02c1ee7596e922afc71c1f3b99bc0677872580ba2ad8716b365 WatchSource:0}: Error finding container 8f3e2a751d55f02c1ee7596e922afc71c1f3b99bc0677872580ba2ad8716b365: Status 404 returned error can't find the container with id 8f3e2a751d55f02c1ee7596e922afc71c1f3b99bc0677872580ba2ad8716b365
Oct 31 04:19:09.102754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.102723    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:09.102828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.102812    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:09.102870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.102841    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:09.686695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.686659    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" event={"ID":"41691951-56fe-4f9a-92ac-2c45175692d3","Type":"ContainerStarted","Data":"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"}
Oct 31 04:19:09.687088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.686724    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" event={"ID":"41691951-56fe-4f9a-92ac-2c45175692d3","Type":"ContainerStarted","Data":"8f3e2a751d55f02c1ee7596e922afc71c1f3b99bc0677872580ba2ad8716b365"}
Oct 31 04:19:09.745281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.745232    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podStartSLOduration=1.7452186269999999 podStartE2EDuration="1.745218627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:09.742958777 +0000 UTC m=+845.775958455" watchObservedRunningTime="2026-10-31 04:19:09.745218627 +0000 UTC m=+845.778218305"
Oct 31 04:19:09.774702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.774672    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:09.890597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:09.890556    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:10.775781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:10.775749    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:10.776230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:10.775799    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:10.776700 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:19:10.776676    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:19:11.775099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:11.774488    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:11.775099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:11.774548    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:11.789072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:11.789032    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:12.620800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:12.620751    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:12.775105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:12.774591    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:12.775105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:12.774692    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:13.615183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.614838    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 04:19:13.639181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.632445    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.652599 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.651527    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-965f61-openshift-default-dockercfg-ggnnv\""
Oct 31 04:19:13.673939 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.673901    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 04:19:13.715181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.705912    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:19:13.715181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.705972    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:19:13.737187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.731810    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.738072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.738038    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm7kx\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.742190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.738277    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.742448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.742428    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.755889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.755854    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.775934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.775892    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.776239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.776215    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.776408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.776390    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.776545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.776530    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.777229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.777150    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:13.777397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.777370    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.877795    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.877846    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm7kx\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.877892    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.877935    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878011    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878058    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878106    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878143    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878200    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.878908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.878665    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.894181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.892612    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.934177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.904206    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.934177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.910530    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.934177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.911728    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:13.947346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.947309    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:19:13.947668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.947639    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:19:13.969032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:13.968998    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:14.002187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.002095    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:14.004110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.004074    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm7kx\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:14.064807 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.064733    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:14.266644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.266495    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:14.795179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.782560    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:14.795179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:14.782619    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:15.209570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.209529    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 04:19:15.318419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.318376    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:15.318702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.318677    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:15.318876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.318856    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:15.509561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.509518    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:19:15.628999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.628525    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:15.774581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.774476    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:15.774842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.774811    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:15.922856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.922819    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" event={"ID":"14645be7-c5dc-4dd0-8518-c29ff76916e4","Type":"ContainerStarted","Data":"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"}
Oct 31 04:19:15.923360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.922885    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" event={"ID":"14645be7-c5dc-4dd0-8518-c29ff76916e4","Type":"ContainerStarted","Data":"cb45975bb674c98d9a8e048f594a3a883c7f390843520d04db20d1781527b63b"}
Oct 31 04:19:15.988182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:15.987576    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podStartSLOduration=2.987556112 podStartE2EDuration="2.987556112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:15.984490557 +0000 UTC m=+852.017490262" watchObservedRunningTime="2026-10-31 04:19:15.987556112 +0000 UTC m=+852.020555794"
Oct 31 04:19:16.268182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:16.267358    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:16.269429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:16.269395    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:16.269547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:16.269470    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:16.774290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:16.774241    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 04:19:16.774481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:16.774320    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 04:19:17.271868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.271738    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:17.274257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.272504    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:17.741196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.736847    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:17.900273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.900242    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:17.900684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.900634    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:17.901567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:17.901550    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:19:18.267468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:18.267430    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:18.267651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:18.267492    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:19.267628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.267590    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:19.268061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.267653    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:19.505876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.505838    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 04:19:19.509820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.509796    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525453    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525503    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525576    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525603    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqj7f\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525745    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525798    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.525851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525845    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.526008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525927    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.526008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.525955    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.560146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.560109    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.626820    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.626887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.626932    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627007    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627055    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627102    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627176    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627213    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.627248    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqj7f\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.630590    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.630889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.630855    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.653178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.631526    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.653448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.646956    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.653585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.647520    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.658263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.658067    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:19.658414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.658312    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.658675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.658653    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.660896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.660871    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqj7f\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.669575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.669544    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:19.849427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:19.849385    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:20.170718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.170628    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 04:19:20.174752 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:19:20.174724    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8d7d49_d719_4fed_971b_89f538d0e39c.slice/crio-786581a7dcca690034295be37d17460d1c9092ff2dd2cd7f819d702ad7904492 WatchSource:0}: Error finding container 786581a7dcca690034295be37d17460d1c9092ff2dd2cd7f819d702ad7904492: Status 404 returned error can't find the container with id 786581a7dcca690034295be37d17460d1c9092ff2dd2cd7f819d702ad7904492
Oct 31 04:19:20.181215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.181181    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:20.181302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.181251    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:20.181302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.181279    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:20.267826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.267796    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:20.268136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.267853    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:20.989918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.989881    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" event={"ID":"df8d7d49-d719-4fed-971b-89f538d0e39c","Type":"ContainerStarted","Data":"631cdf3d5ed9ad348cfe082751199772c28f3c2ab917ff36351ca9fd79abb941"}
Oct 31 04:19:20.990125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:20.989939    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" event={"ID":"df8d7d49-d719-4fed-971b-89f538d0e39c","Type":"ContainerStarted","Data":"786581a7dcca690034295be37d17460d1c9092ff2dd2cd7f819d702ad7904492"}
Oct 31 04:19:21.086685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.086615    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podStartSLOduration=2.086595531 podStartE2EDuration="2.086595531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:21.086401002 +0000 UTC m=+857.119400682" watchObservedRunningTime="2026-10-31 04:19:21.086595531 +0000 UTC m=+857.119595211"
Oct 31 04:19:21.271240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.270621    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:21.271240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.270689    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:21.663182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.662774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:21.853182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.852554    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:21.870217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.864298    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:21.870217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:21.864360    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:22.267453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:22.267411    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 04:19:22.267617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:22.267482    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 04:19:22.852181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:22.851626    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:22.852181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:22.851684    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:23.374032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.373996    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:23.374502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.374472    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:23.375690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.375668    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 04:19:23.657848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.657729    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:23.851746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.851705    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:23.852132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:23.852079    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:24.851691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:24.851645    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:24.852085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:24.851732    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:25.640182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:25.639648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:25.851635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:25.851597    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:25.851857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:25.851658    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:26.851111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:26.851071    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 04:19:26.851298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:26.851148    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 04:19:27.289122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:27.289080    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:27.748425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:27.748330    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:28.913383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:28.913331    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:19:28.914054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:28.914022    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:19:29.068291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:29.068258    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:29.068717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:29.068688    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:29.070735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:29.070710    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 04:19:29.987140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:29.987087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:31.894654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:31.894607    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:33.717841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:33.717804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:34.555628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.555594    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:19:34.559932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.559911    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.589178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.589132    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:19:34.711300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711262    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711315    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfh4\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711415    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711454    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711548    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711578    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711599    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711725    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.711805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.711789    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813169    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813203    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813223    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813240    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813365    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813485    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813568    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813595    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813619    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813659    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813708    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfh4\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.813861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813817    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.814327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.813862    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.814407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.814386    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.815947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.815919    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.816040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.815922    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.835621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.835597    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.836377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.836359    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfh4\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\") pod \"e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") " pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:34.875390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:34.875361    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:35.195348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:35.195190    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:19:35.197960 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:19:35.197934    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711397bd_dfac_4257_9fcc_54cfa1daba92.slice/crio-20431ee0023d39ec3cadf0d317b58be1b992d5b851450ab71ec997085459ea75 WatchSource:0}: Error finding container 20431ee0023d39ec3cadf0d317b58be1b992d5b851450ab71ec997085459ea75: Status 404 returned error can't find the container with id 20431ee0023d39ec3cadf0d317b58be1b992d5b851450ab71ec997085459ea75
Oct 31 04:19:35.200547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:35.200514    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:35.200629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:35.200581    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:35.200629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:35.200613    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:19:35.636341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:35.636306    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:36.083314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.082636    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" event={"ID":"711397bd-dfac-4257-9fcc-54cfa1daba92","Type":"ContainerStarted","Data":"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"}
Oct 31 04:19:36.083314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.082697    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" event={"ID":"711397bd-dfac-4257-9fcc-54cfa1daba92","Type":"ContainerStarted","Data":"20431ee0023d39ec3cadf0d317b58be1b992d5b851450ab71ec997085459ea75"}
Oct 31 04:19:36.152182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.151203    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podStartSLOduration=2.151181231 podStartE2EDuration="2.151181231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:36.148723011 +0000 UTC m=+872.181722714" watchObservedRunningTime="2026-10-31 04:19:36.151181231 +0000 UTC m=+872.184180912"
Oct 31 04:19:36.875973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.875935    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:36.877604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.877573    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:36.877709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:36.877618    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:37.875972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:37.875923    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:37.876401 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:37.875996    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:37.951429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:37.951373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:38.876721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:38.876679    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:38.877268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:38.876732    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:39.876722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:39.876685    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:39.877039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:39.876746    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:40.285044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:40.285009    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:40.777236 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:19:40.777139    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:19:40.876773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:40.876738    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:40.876964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:40.876789    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:41.267129 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.267087    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 04:19:41.272085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.272057    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.291145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.291117    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 04:19:41.298867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.298734    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"openshift-service-ca.crt\""
Oct 31 04:19:41.298867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.298784    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"distinct-2-w3-4c2387-kserve-self-signed-certs\""
Oct 31 04:19:41.298867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.298806    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"kube-root-ca.crt\""
Oct 31 04:19:41.298867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.298830    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4c2387\"/\"default-dockercfg-zfb5b\""
Oct 31 04:19:41.302668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302644    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.302780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302687    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.302780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302714    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vhf\" (UniqueName: \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.302871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302808    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.302871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302865    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.302990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.302969    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404493 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404453    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404515    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404617    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404648    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404669    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vhf\" (UniqueName: \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.404920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404714    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.405019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.404993    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.405085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.405042    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.405170 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.405079    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.407528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.407502    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.408260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.408230    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.423900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.423864    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:41.430797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.430764    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vhf\" (UniqueName: \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\") pod \"distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") " pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.588171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.588121    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:41.875916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.875836    6931 patch_prober.go:29] interesting pod/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 04:19:41.875916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.875886    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 04:19:41.903418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:41.903375    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 04:19:41.907481 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:19:41.907454    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod581d4326_a691_421a_8f6b_3da791d651dd.slice/crio-df2aba72dfa25f2ba9940d071a8aefa74d20208221f8d0ae4a0ef9afd0fef5fc WatchSource:0}: Error finding container df2aba72dfa25f2ba9940d071a8aefa74d20208221f8d0ae4a0ef9afd0fef5fc: Status 404 returned error can't find the container with id df2aba72dfa25f2ba9940d071a8aefa74d20208221f8d0ae4a0ef9afd0fef5fc
Oct 31 04:19:42.124212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.124148    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" event={"ID":"581d4326-a691-421a-8f6b-3da791d651dd","Type":"ContainerStarted","Data":"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"}
Oct 31 04:19:42.124212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.124221    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" event={"ID":"581d4326-a691-421a-8f6b-3da791d651dd","Type":"ContainerStarted","Data":"df2aba72dfa25f2ba9940d071a8aefa74d20208221f8d0ae4a0ef9afd0fef5fc"}
Oct 31 04:19:42.166721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.166597    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" podStartSLOduration=1.166579309 podStartE2EDuration="1.166579309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:19:42.164276732 +0000 UTC m=+878.197276436" watchObservedRunningTime="2026-10-31 04:19:42.166579309 +0000 UTC m=+878.199578990"
Oct 31 04:19:42.359357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.359309    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:42.588798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.588756    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:42.880418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.880350    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:42.880718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.880693    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:42.881399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:42.881383    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:19:44.214859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:44.214822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:46.088554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:46.088512    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:48.517178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:48.516549    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:51.588316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:51.588280    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:51.601109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:51.601083    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:52.267139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:52.267111    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:19:52.641429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:52.641379    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:54.499644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:54.499606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:56.379169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:56.379130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:56.593436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:56.593396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:58.522500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:58.522454    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:59.202101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:59.202070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:59.739890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:19:59.739854    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:00.022086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:00.021998    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:00.688080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:00.688043    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:02.659188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:02.659108    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:03.384667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:03.384626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:03.597383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:03.597350    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:04.654578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:04.654538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:05.118378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:05.118341    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:05.337653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:05.337600    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:06.540373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:06.540337    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:06.774619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:06.774586    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:07.763471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:07.763435    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:07.979321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:07.979276    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:08.619106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:08.619071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:10.950188 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:10.948936    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:20:11.348307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:11.348251    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:11.524336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:11.524287    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:12.379187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:12.376143    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:12.970881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:12.970849    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:20:12.971452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:12.971434    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:20:13.799079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.799041    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:20:13.804296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.804273    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.807597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.807570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:13.839318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.839281    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:20:13.842229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.842201    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:20:13.847122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.847100    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.872962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.872919    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:20:13.910568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910529    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.910758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910578    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.910758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910701    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.910758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910740    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.910916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910853    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.910972    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911010    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911050    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911125    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911249    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911328 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911301    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911362    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911404    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:13.911451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911433    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6jv\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911506    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911543    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911571    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:13.911666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:13.911603    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.012947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.012909    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.012953    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.012985    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013013    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013059    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013090    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013133    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013192    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013231    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013257    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013281    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013307    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013347    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013396    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013450    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013493    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013538    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013559    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013570    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6jv\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013852    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013865    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013920    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.013985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.013924    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.014390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.014126    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.014390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.014232    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.014942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.014407    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.014942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.014450    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.014942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.014709    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.016087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.016062    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.016204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.016082    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.016264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.016210    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.016357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.016340    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.038080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.038049    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.038234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.038100    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.038287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.038270    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6jv\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-g99lj\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.038324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.038292    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.119722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.119621    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:14.195447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.195414    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:14.444224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.444136    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:20:14.447676 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:14.447647    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a48553f_8d2f_48c6_b2f1_3d3e359c8d99.slice/crio-6683ce04c2f6c9d4d5dddcf3a90b1eafe2b4dc3e419ea97bbfd7c5c3c559ebaa WatchSource:0}: Error finding container 6683ce04c2f6c9d4d5dddcf3a90b1eafe2b4dc3e419ea97bbfd7c5c3c559ebaa: Status 404 returned error can't find the container with id 6683ce04c2f6c9d4d5dddcf3a90b1eafe2b4dc3e419ea97bbfd7c5c3c559ebaa
Oct 31 04:20:14.450008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.449970    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.450119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.450051    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.450119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.450098    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.502372 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.502307    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:20:14.507050 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:14.507019    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df059bf_ad76_46b7_b975_2a71cc4d2aed.slice/crio-292e1d3fa66f175aee51f2a325b11ab675dab61d0b9431065b9aa1d3f9ec7611 WatchSource:0}: Error finding container 292e1d3fa66f175aee51f2a325b11ab675dab61d0b9431065b9aa1d3f9ec7611: Status 404 returned error can't find the container with id 292e1d3fa66f175aee51f2a325b11ab675dab61d0b9431065b9aa1d3f9ec7611
Oct 31 04:20:14.509566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.509533    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.509651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.509599    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.509651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.509640    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:14.823348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.823309    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" event={"ID":"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99","Type":"ContainerStarted","Data":"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"}
Oct 31 04:20:14.823818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.823379    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" event={"ID":"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99","Type":"ContainerStarted","Data":"6683ce04c2f6c9d4d5dddcf3a90b1eafe2b4dc3e419ea97bbfd7c5c3c559ebaa"}
Oct 31 04:20:14.825059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.825037    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" event={"ID":"0df059bf-ad76-46b7-b975-2a71cc4d2aed","Type":"ContainerStarted","Data":"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"}
Oct 31 04:20:14.825172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.825075    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" event={"ID":"0df059bf-ad76-46b7-b975-2a71cc4d2aed","Type":"ContainerStarted","Data":"292e1d3fa66f175aee51f2a325b11ab675dab61d0b9431065b9aa1d3f9ec7611"}
Oct 31 04:20:14.868110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.868057    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podStartSLOduration=1.868041466 podStartE2EDuration="1.868041466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:14.867692029 +0000 UTC m=+910.900691710" watchObservedRunningTime="2026-10-31 04:20:14.868041466 +0000 UTC m=+910.901041146"
Oct 31 04:20:14.915989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:14.915923    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podStartSLOduration=1.915900659 podStartE2EDuration="1.915900659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:14.911898299 +0000 UTC m=+910.944897979" watchObservedRunningTime="2026-10-31 04:20:14.915900659 +0000 UTC m=+910.948900339"
Oct 31 04:20:15.120819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.120743    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:15.121964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.121940    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:15.122014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.121983    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:15.196766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.196723    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:15.197855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.197811    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:15.197981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:15.197869    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:16.120551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:16.120517    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:16.120942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:16.120565    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:16.195833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:16.195793    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:16.196006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:16.195845    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:16.488917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:16.488822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:17.120254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:17.120217    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:17.120454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:17.120279    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:17.195862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:17.195821    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:17.196268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:17.195879    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:18.120669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.120639    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:18.120887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.120713    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:18.196613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.196487    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:18.196613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.196546    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:18.198106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.198079    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:20:18.202792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.202771    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.218930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.218904    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-146320d5-serving-cert\""
Oct 31 04:20:18.287925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.287879    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:20:18.315102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.314999    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.315102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.315037    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4rf\" (UniqueName: \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.315102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.315056    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.315507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.315279    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.396929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.396855    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:18.416741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.416710    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.417314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.417299    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4rf\" (UniqueName: \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.417744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.417730    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.418121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.418108    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.418867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.418682    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.419698 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.419678    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.419937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.419915    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.453887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.453854    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4rf\" (UniqueName: \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\") pod \"maas-api-e2e-mt-146320d5-8478684b9-t8m4x\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.520364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.520333    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:18.611787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.611735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:18.980668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.980577    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:20:18.985842 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:18.985818    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdf93d6_0c51_45e6_8a7b_cbc0e9d32e99.slice/crio-b0b29223ae2e3a9afcf242420c8ea3d857d1c0538481648b352e61a33af6812a WatchSource:0}: Error finding container b0b29223ae2e3a9afcf242420c8ea3d857d1c0538481648b352e61a33af6812a: Status 404 returned error can't find the container with id b0b29223ae2e3a9afcf242420c8ea3d857d1c0538481648b352e61a33af6812a
Oct 31 04:20:18.987612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:18.987596    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:20:19.120046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.119986    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:19.120216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.120066    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:19.196882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.196655    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:19.196882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.196735    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:19.903369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.903338    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:19.908148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.908118    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" event={"ID":"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99","Type":"ContainerStarted","Data":"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"}
Oct 31 04:20:19.908320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.908185    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" event={"ID":"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99","Type":"ContainerStarted","Data":"b0b29223ae2e3a9afcf242420c8ea3d857d1c0538481648b352e61a33af6812a"}
Oct 31 04:20:19.908320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.908211    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:19.955181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:19.954639    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" podStartSLOduration=1.509367874 podStartE2EDuration="1.954617315s" totalImagesPullingTime="445.249441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:19.951761703 +0000 UTC m=+915.984761408" watchObservedRunningTime="2026-10-31 04:20:19.954617315 +0000 UTC m=+915.987616996"
Oct 31 04:20:20.122693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:20.122662    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:20.122900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:20.122881    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:20.203409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:20.203322    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:20.203987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:20.203956    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:20.644183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:20.636765    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:21.120607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.120550    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 04:20:21.120793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.120656    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 04:20:21.196122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.196075    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 04:20:21.196336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.196134    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 04:20:21.237499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.237462    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 04:20:21.241959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.241934    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.276326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.276294    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 04:20:21.317872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.317813    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.317971    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318080    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318120    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318168    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l56j\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318198    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318235    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318337    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.318556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.318435    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.419751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.419664    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.419751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.419732    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.419754    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.419784    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l56j\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.419840    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420196    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420231    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420258    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420279    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420285    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420381    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420698    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.420766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.420715    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.421275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.421250    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.422571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.422542    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.422759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.422739    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.443756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.443722    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l56j\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.444938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.444911    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.561452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.561415    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:21.898554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.898080    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 04:20:21.907355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.907313    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:21.907468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.907406    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:21.907468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.907456    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:21.926448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:21.926406    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" event={"ID":"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2","Type":"ContainerStarted","Data":"16f72b1bf253cf4ead32414e44ba1cdc780f48394e21f3752b7111e2114acfa3"}
Oct 31 04:20:22.162494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.162407    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:22.163344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.163324    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:22.164528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.164501    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:20:22.237933 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.237886    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:22.238454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.238225    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:22.239212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.239193    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:20:22.523879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.523780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:22.956792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:22.956757    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" event={"ID":"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2","Type":"ContainerStarted","Data":"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"}
Oct 31 04:20:23.001329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.001268    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podStartSLOduration=2.001251384 podStartE2EDuration="2.001251384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:22.999958707 +0000 UTC m=+919.032958406" watchObservedRunningTime="2026-10-31 04:20:23.001251384 +0000 UTC m=+919.034251079"
Oct 31 04:20:23.561784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.561747    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:23.563420 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.563398    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:23.563521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.563443    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:23.713929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.713898    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:20:23.718168 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.718138    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.762741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.762702    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:20:23.765416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765385    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765446    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2nh\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765520    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765562    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765619    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765657    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.765748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765715    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.767605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765779    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.767605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.765816    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.772717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.772687    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:20:23.778814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.778784    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:20:23.778986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.778897    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.866481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866391    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.866481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866440    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.866481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866469    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.866708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866499    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.866793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866772    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.866855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866812    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.866912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866895    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.866973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866930    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.866973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866958    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.866987    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867016    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867053    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867107    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867134    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867201    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjvn\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867397    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2nh\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867519    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867548    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.867731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867592    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867623    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867737    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867881    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.867965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.867951    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.868847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.868800    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.871439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.871392    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.889964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.889929    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.899906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.899870    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2nh\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:23.968350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968315    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968371    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968426    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968442    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968469    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968814    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968827    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968854    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjvn\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.968970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.968901    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.969080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.969007    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.969136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.969104    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.969136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.969129    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.969256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.969137    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.969314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.969264    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.971337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.971315    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.971510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.971492    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.991893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.991862    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:23.991987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:23.991945    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjvn\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:24.064991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.064951    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:24.104127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.104028    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:24.413778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.413745    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:20:24.419755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.419713    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.419883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.419793    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.419883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.419831    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.472431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.472400    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:20:24.478055 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:24.478020    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abc0e64_26b1_4bdb_80bd_27ee4445ad4d.slice/crio-04fc21f6d90fb9be4c62d4c308d17dff063761d208e265bd3613a409641cb5df WatchSource:0}: Error finding container 04fc21f6d90fb9be4c62d4c308d17dff063761d208e265bd3613a409641cb5df: Status 404 returned error can't find the container with id 04fc21f6d90fb9be4c62d4c308d17dff063761d208e265bd3613a409641cb5df
Oct 31 04:20:24.481071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.481037    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.481181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.481108    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.481181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.481137    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:24.567213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.565689    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:24.567213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.565754    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:24.567814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.567611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:24.972044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.971947    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" event={"ID":"87ba26d8-2892-4a05-b8b9-5ad452474494","Type":"ContainerStarted","Data":"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"}
Oct 31 04:20:24.972044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.972005    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" event={"ID":"87ba26d8-2892-4a05-b8b9-5ad452474494","Type":"ContainerStarted","Data":"91e8561fed5ae728ed44c454e3036ea7d4e71f68139ea077f81946a30482d7e8"}
Oct 31 04:20:24.974031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.974001    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" event={"ID":"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d","Type":"ContainerStarted","Data":"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"}
Oct 31 04:20:24.974199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:24.974065    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" event={"ID":"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d","Type":"ContainerStarted","Data":"04fc21f6d90fb9be4c62d4c308d17dff063761d208e265bd3613a409641cb5df"}
Oct 31 04:20:25.018362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.018192    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podStartSLOduration=2.018174537 podStartE2EDuration="2.018174537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:25.016261456 +0000 UTC m=+921.049261160" watchObservedRunningTime="2026-10-31 04:20:25.018174537 +0000 UTC m=+921.051174229"
Oct 31 04:20:25.065558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.065525    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:25.069231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.069175    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podStartSLOduration=2.069143565 podStartE2EDuration="2.069143565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:25.067441446 +0000 UTC m=+921.100441126" watchObservedRunningTime="2026-10-31 04:20:25.069143565 +0000 UTC m=+921.102143245"
Oct 31 04:20:25.071275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.071247    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:25.071399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.071299    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:25.110406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.110370    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:25.112678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.112648    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:25.112815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.112699    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:25.568181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.567725    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:25.568181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.567786    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:25.704787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.704629    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:25.924635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:25.924553    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:20:26.125181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.124348    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:26.125181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.124409    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:26.248332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.225701    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:26.248332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.225769    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:26.265206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.251833    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:26.568182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.564013    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:26.568182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.564075    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:26.568182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.565758    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:26.993240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:26.992181    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:27.069205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.068575    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:27.069205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.068651    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:27.106182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.105593    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:27.106182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.105659    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:27.574050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.573384    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:27.574050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.573445    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:27.610184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:27.608865    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:28.071646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.071607    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:28.071895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.071870    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:28.111462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.111424    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:28.127477 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.125281    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:28.564538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.564497    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 04:20:28.564838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:28.564807    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 04:20:29.067637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.067596    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:29.068275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.068242    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:29.104408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.104374    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:29.104585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.104427    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:29.353237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.352684    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:29.755226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.755125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:29.978273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:29.978228    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:30.067590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.067557    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:30.068697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.068666    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:30.106336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.106293    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:30.106477 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.106363    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:30.186585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.186553    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:30.187785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.187764    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:30.188635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.188610    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 04:20:30.257300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.252474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:30.794925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:30.794851    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:31.014671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.014637    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:31.066087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.065573    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 04:20:31.066087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.065632    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 04:20:31.105091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.105054    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 04:20:31.105560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.105122    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 04:20:31.611498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.611459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:31.960386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:31.960295    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:32.267832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.261723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:32.512362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.512330    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:32.512904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.512885    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:32.513857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.513830    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:20:32.618612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.618576    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:32.619036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.619012    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:32.620026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:32.620006    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:20:33.699887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:33.699839    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:35.045552 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:35.045517    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:35.264847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:35.264788    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:35.810632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:35.810595    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:37.292215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:37.292178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:37.549535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:37.549424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:38.059675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.059634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:38.758528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.758498    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:20:38.762753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.762733    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.778295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.778266    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4c2387-openshift-default-dockercfg-x4j6w\""
Oct 31 04:20:38.794022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.793984    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:20:38.841512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.841471    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:38.874854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.874816    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.874854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.874853    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6df4d\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.874994    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875036    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875090    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875121    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875259    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875343    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.875393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.875376    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976531    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976574    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976605    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976626    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6df4d\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976713    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976755    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976785    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976817    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.976990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.976881    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.977185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.977077    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.977267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.977245    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.977598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.977367    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.977598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.977398    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.977598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.977558    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.978956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.978934    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.979209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.979191    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.999805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.999777    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:38.999968 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:38.999854    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6df4d\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:39.081718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:39.081685    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:39.408000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:39.407913    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:20:39.413274 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:39.413244    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244dbe89_ede1_4972_8e08_185804450fa2.slice/crio-1f87f188cbf78c2db6d1d1b320f4c66a230f5948770c125ef980d3eaf0af9d5d WatchSource:0}: Error finding container 1f87f188cbf78c2db6d1d1b320f4c66a230f5948770c125ef980d3eaf0af9d5d: Status 404 returned error can't find the container with id 1f87f188cbf78c2db6d1d1b320f4c66a230f5948770c125ef980d3eaf0af9d5d
Oct 31 04:20:39.415991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:39.415955    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:39.416100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:39.416051    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:39.416203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:39.416101    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:40.193539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:40.193493    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" event={"ID":"244dbe89-ede1-4972-8e08-185804450fa2","Type":"ContainerStarted","Data":"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"}
Oct 31 04:20:40.194440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:40.194413    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" event={"ID":"244dbe89-ede1-4972-8e08-185804450fa2","Type":"ContainerStarted","Data":"1f87f188cbf78c2db6d1d1b320f4c66a230f5948770c125ef980d3eaf0af9d5d"}
Oct 31 04:20:40.292181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:40.280180    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podStartSLOduration=2.2801414749999998 podStartE2EDuration="2.280141475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:40.250582102 +0000 UTC m=+936.283581810" watchObservedRunningTime="2026-10-31 04:20:40.280141475 +0000 UTC m=+936.313141155"
Oct 31 04:20:40.986182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:40.985949    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:41.033445 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:41.033366    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:20:41.087021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:41.086936    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:41.088743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:41.088719    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:41.088947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:41.088916    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:41.688180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:41.680081    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:42.082700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.082663    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:42.083023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.082975    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:42.099888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.099848    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:42.257324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.257280    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:42.318110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.318075    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:42.336424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.336299    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:42.541826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:42.541771    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:43.151202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.134302    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 04:20:43.328086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.196795    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:43.330765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.330726    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:43.330964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.270517    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:43.331422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.331398    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:20:43.353804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.353759    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:20:43.355543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.354423    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:20:43.422288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.420350    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 04:20:43.552307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.552260    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 04:20:43.553043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:43.553021    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:20:44.029218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:44.026743    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:20:44.095267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:44.095228    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:44.097271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:44.097230    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:44.235280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:44.225656    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:44.406729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:44.404476    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 04:20:45.082617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.082583    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:45.082812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.082636    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:45.863187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.863122    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:20:45.868451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.868432    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.880903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.880881    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8c0086-serving-cert\""
Oct 31 04:20:45.899197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.894654    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:20:45.899197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.896176    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.899197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.896283    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.899197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.896555    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.899197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.896600    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f57m\" (UniqueName: \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.997581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.997539    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.997581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.997589    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f57m\" (UniqueName: \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.997831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.997632    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.997831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.997683    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:45.997899 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:45.997826    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-serving-cert: secret "maas-api-e2e-api-a-8c0086-serving-cert" not found
Oct 31 04:20:45.997899 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:45.997883    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls podName:316c7c51-4157-4df8-8dfa-7bceb6238555 nodeName:}" failed. No retries permitted until 2026-10-31 04:20:46.497867084 +0000 UTC m=+942.530866741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls") pod "maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555") : secret "maas-api-e2e-api-a-8c0086-serving-cert" not found
Oct 31 04:20:45.998342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:45.998326    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:46.000064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.000044    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:46.019259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.019227    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f57m\" (UniqueName: \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:46.056432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.056284    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:46.081992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.081957    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:46.082183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.082029    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:46.268866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.268780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:46.504439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:46.504402    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:46.504609 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:46.504556    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-serving-cert: secret "maas-api-e2e-api-a-8c0086-serving-cert" not found
Oct 31 04:20:46.504660 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:20:46.504618    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls podName:316c7c51-4157-4df8-8dfa-7bceb6238555 nodeName:}" failed. No retries permitted until 2026-10-31 04:20:47.504602318 +0000 UTC m=+943.537601996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls") pod "maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555") : secret "maas-api-e2e-api-a-8c0086-serving-cert" not found
Oct 31 04:20:47.084178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.083134    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:47.084178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.083210    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:47.315350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.315307    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:47.523558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.523467    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:47.525938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.525912    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:47.682716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:47.682679    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:48.001437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.001258    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:20:48.011780 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:48.011741    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316c7c51_4157_4df8_8dfa_7bceb6238555.slice/crio-818a55fa4bcdafb48ee7948d6c4fe0aa426159a469cc731dcafff20a38467577 WatchSource:0}: Error finding container 818a55fa4bcdafb48ee7948d6c4fe0aa426159a469cc731dcafff20a38467577: Status 404 returned error can't find the container with id 818a55fa4bcdafb48ee7948d6c4fe0aa426159a469cc731dcafff20a38467577
Oct 31 04:20:48.082382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.082346    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 04:20:48.082581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.082403    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 04:20:48.110827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.110782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:48.441690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.441638    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" event={"ID":"316c7c51-4157-4df8-8dfa-7bceb6238555","Type":"ContainerStarted","Data":"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"}
Oct 31 04:20:48.442026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.442007    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:48.442260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.442235    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" event={"ID":"316c7c51-4157-4df8-8dfa-7bceb6238555","Type":"ContainerStarted","Data":"818a55fa4bcdafb48ee7948d6c4fe0aa426159a469cc731dcafff20a38467577"}
Oct 31 04:20:48.485058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:48.484999    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" podStartSLOduration=3.168183978 podStartE2EDuration="3.484985363s" totalImagesPullingTime="316.801385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:48.482741956 +0000 UTC m=+944.515741636" watchObservedRunningTime="2026-10-31 04:20:48.484985363 +0000 UTC m=+944.517985044"
Oct 31 04:20:49.086711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.086682    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:49.087048 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.087013    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:49.087788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.087763    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:20:49.483909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.483835    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 04:20:49.487845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.487827    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.502485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.502434    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 04:20:49.502485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.502438    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 04:20:49.502677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.502495    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 04:20:49.502677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.502598    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m4kc9\""
Oct 31 04:20:49.518136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.518105    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 04:20:49.548712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.548670    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.548881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.548720    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.548976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.548946    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.549039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.548991    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.549134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.549115    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.549241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.549187    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.549308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.549279    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjmm\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650078    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650135    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650203    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjmm\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650300    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650321    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650387    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.650487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.650415    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.651199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.651153    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.652748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.652722    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.652748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.652736    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.652915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.652876    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.652981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.652914    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.675797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.675767    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.675962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.675827    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjmm\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7wbkm\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:49.801145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:49.801110    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:50.005413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:50.005378    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:50.124340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:50.124269    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 04:20:50.127884 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:20:50.127859    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb7d9e7_1ecc_4b76_97d3_770935fe5039.slice/crio-29dbbcfa937af098a9e02d4fdc395ec5d5bec3f63e58529f6d113117c032e642 WatchSource:0}: Error finding container 29dbbcfa937af098a9e02d4fdc395ec5d5bec3f63e58529f6d113117c032e642: Status 404 returned error can't find the container with id 29dbbcfa937af098a9e02d4fdc395ec5d5bec3f63e58529f6d113117c032e642
Oct 31 04:20:50.455131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:50.455056    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" event={"ID":"4bb7d9e7-1ecc-4b76-97d3-770935fe5039","Type":"ContainerStarted","Data":"29dbbcfa937af098a9e02d4fdc395ec5d5bec3f63e58529f6d113117c032e642"}
Oct 31 04:20:51.859437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:51.857766    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:52.070663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:52.070618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:52.285786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:52.285754    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:52.841183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:52.841132    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:52.841285 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:52.841214    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:20:53.480961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.480922    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" event={"ID":"4bb7d9e7-1ecc-4b76-97d3-770935fe5039","Type":"ContainerStarted","Data":"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"}
Oct 31 04:20:53.481461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.481105    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:53.483528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.483501    6931 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7wbkm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:20:53.483667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.483555    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:20:53.552100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.552046    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" podStartSLOduration=1.841880362 podStartE2EDuration="4.552032165s" totalImagesPullingTime="2.710151803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:53.537030957 +0000 UTC m=+949.570030656" watchObservedRunningTime="2026-10-31 04:20:53.552032165 +0000 UTC m=+949.585031844"
Oct 31 04:20:53.844785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:53.844741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:54.489857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:54.489818    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 04:20:55.461186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:55.461126    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:20:55.724628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:55.724541    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:56.511807 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:56.511770    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:57.516031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:57.515987    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:59.307208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:59.307129    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:59.566982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:20:59.566898    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:01.251572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:01.251529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:03.106870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:03.106799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:05.076171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:05.076119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:07.003078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:07.003042    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:08.924736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:08.924654    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:10.382999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:10.382953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:10.761277 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:21:10.761189    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:21:11.160296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:11.160255    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:11.679731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:11.679691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:11.887956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:11.887874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:12.165424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:12.165385    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:14.986650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:14.986617    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:15.643746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:15.643707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:16.048776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:16.048743    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:16.258328 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:16.258296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:16.483222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:16.483082    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:16.928379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:16.928273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:17.222034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:17.221930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:17.703098 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:17.703062    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:18.089018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:18.088976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:18.339343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:18.339242    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:19.943987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:19.943945    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:20.675839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:20.675806    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:20.889223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:20.889190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:22.097589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:22.097537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:22.498738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:22.498643    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:22.743169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:22.743119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:24.423073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:24.423027    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:27.212543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:27.212501    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:29.627549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:29.627516    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:31.473023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:31.472991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:33.231338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:33.230869    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:35.066957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:35.066888    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:35.281256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:35.281212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:35.495514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:35.495423    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:37.108127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:37.108097    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:38.941785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:38.941742    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:38.945332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:38.945306    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:21:38.950854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:38.950833    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:38.979249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:38.979210    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:21:39.143964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.143912    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.143987    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144027    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mk2\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144131    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144170    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144203    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144287    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144341    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.144431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.144395    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.245949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.245850    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.245949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.245915    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.245949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.245938    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mk2\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.245949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.245960    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.245994    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246031    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246112    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246199    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246229    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246483    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246592    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246790 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246768    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246842    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.246983 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.246960    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.248373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.248352    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.248514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.248496    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.268808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.268776    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.268894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.268782    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mk2\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.567056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.567018    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:39.890996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.890918    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:21:39.894657 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:21:39.894624    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba4a620_55af_4497_9549_a21577316dec.slice/crio-7debc97eaa4241949b422d773e347294fc8912e58ecea3380958b416e6d11962 WatchSource:0}: Error finding container 7debc97eaa4241949b422d773e347294fc8912e58ecea3380958b416e6d11962: Status 404 returned error can't find the container with id 7debc97eaa4241949b422d773e347294fc8912e58ecea3380958b416e6d11962
Oct 31 04:21:39.897116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.897080    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:21:39.897241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.897149    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:21:39.897241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:39.897198    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:21:40.566914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.566881    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:21:40.570953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.570935    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.588328 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.588303    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a43a70-serving-cert\""
Oct 31 04:21:40.605427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.605391    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:21:40.666678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.666631    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.666880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.666722    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.666880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.666797    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.666880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.666861    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vp7\" (UniqueName: \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.768629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.768562    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.769052 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.768641    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.769052 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.768670    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.769052 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.768706    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vp7\" (UniqueName: \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.769375 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:21:40.769352    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-serving-cert: secret "maas-api-e2e-api-b-a43a70-serving-cert" not found
Oct 31 04:21:40.769580 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:21:40.769565    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls podName:0d129d68-c6f1-42dd-af60-11c02795999e nodeName:}" failed. No retries permitted until 2026-10-31 04:21:41.269539325 +0000 UTC m=+997.302538998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls") pod "maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" (UID: "0d129d68-c6f1-42dd-af60-11c02795999e") : secret "maas-api-e2e-api-b-a43a70-serving-cert" not found
Oct 31 04:21:40.769936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.769913    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.780999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.780960    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.791965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.791912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:40.795760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.795728    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vp7\" (UniqueName: \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:40.797588 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:21:40.797556    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:21:40.882043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.881948    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" event={"ID":"6ba4a620-55af-4497-9549-a21577316dec","Type":"ContainerStarted","Data":"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"}
Oct 31 04:21:40.882043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.882035    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" event={"ID":"6ba4a620-55af-4497-9549-a21577316dec","Type":"ContainerStarted","Data":"7debc97eaa4241949b422d773e347294fc8912e58ecea3380958b416e6d11962"}
Oct 31 04:21:40.935185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:40.928382    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podStartSLOduration=2.928361866 podStartE2EDuration="2.928361866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:40.926258993 +0000 UTC m=+996.959258693" watchObservedRunningTime="2026-10-31 04:21:40.928361866 +0000 UTC m=+996.961361547"
Oct 31 04:21:41.017509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.017478    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:41.252784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.252693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:41.275591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.275561    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:41.277971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.277948    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") pod \"maas-api-e2e-api-b-a43a70-846dc884c6-vsthm\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:41.485205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.485164    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:41.571835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.569612    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:41.571835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.571494    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:41.571835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.571559    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:41.868151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.868072    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:21:41.871932 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:21:41.871905    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d129d68_c6f1_42dd_af60_11c02795999e.slice/crio-4f5c6c75e2ddba3792ec2f2f73326c9c4b246f006ca212a24cb5fbe6cf0cce28 WatchSource:0}: Error finding container 4f5c6c75e2ddba3792ec2f2f73326c9c4b246f006ca212a24cb5fbe6cf0cce28: Status 404 returned error can't find the container with id 4f5c6c75e2ddba3792ec2f2f73326c9c4b246f006ca212a24cb5fbe6cf0cce28
Oct 31 04:21:41.907868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:41.907827    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" event={"ID":"0d129d68-c6f1-42dd-af60-11c02795999e","Type":"ContainerStarted","Data":"4f5c6c75e2ddba3792ec2f2f73326c9c4b246f006ca212a24cb5fbe6cf0cce28"}
Oct 31 04:21:42.567864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.567827    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:42.567976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.567887    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:42.890822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.890748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:42.915506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.915467    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" event={"ID":"0d129d68-c6f1-42dd-af60-11c02795999e","Type":"ContainerStarted","Data":"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"}
Oct 31 04:21:42.915709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.915534    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:42.974580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:42.974515    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" podStartSLOduration=2.657393643 podStartE2EDuration="2.974495999s" totalImagesPullingTime="317.102356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:42.974284623 +0000 UTC m=+999.007284321" watchObservedRunningTime="2026-10-31 04:21:42.974495999 +0000 UTC m=+999.007495682"
Oct 31 04:21:43.568298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:43.568258    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:43.568483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:43.568328    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:44.567828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:44.567750    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:44.567828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:44.567804    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:44.891215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:44.891115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:45.567853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:45.567814    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:45.568289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:45.567870    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:46.568412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:46.568376    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 04:21:46.568910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:46.568428    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 04:21:46.900778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:46.900690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:47.576463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:47.576431    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:47.590184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:47.587373    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:47.594408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:47.594383    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:21:48.929134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:48.929105    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:21:49.131771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:49.131732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:51.230114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:51.230079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:53.115974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:53.115937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:55.012657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:55.012622    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:56.648190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:56.648126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:56.867482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:56.867419    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:58.640037 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:21:58.640007    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:00.512204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:00.512127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:01.025685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:01.025643    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:01.793504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:01.793417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:02.408116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:02.408074    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:04.500093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:04.500051    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:04.718345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:04.718301    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:04.931502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:04.931464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:06.424059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:06.424008    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:06.637961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:06.637926    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:07.490184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:07.490119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:08.230974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:08.230937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:08.446111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:08.446069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:10.203087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:10.203047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:10.438217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:10.438182    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:10.644174 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:10.644133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:10.760601 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:22:10.760567    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:22:11.104290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:11.104250    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:12.442173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:12.442125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:14.314981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:14.314945    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:14.571239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:14.563843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:14.775192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:14.775120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:15.311492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:15.311440    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:16.946917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:16.946825    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:17.303189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:17.302823    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:18.965318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:18.965281    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:19.178784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:19.178741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:19.382127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:19.382079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:21.367393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:21.367357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:21.580755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:21.580728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:23.499501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:23.499464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:25.401058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:25.401023    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:27.300497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:27.300453    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:29.200021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:29.199982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:30.615818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:30.615775    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:22:30.616690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:30.616020    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" podUID="316c7c51-4157-4df8-8dfa-7bceb6238555" containerName="maas-api" containerID="cri-o://0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe" gracePeriod=30
Oct 31 04:22:30.789423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:30.789379    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:30.988275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:30.987929    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:22:31.162651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.162619    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\") pod \"316c7c51-4157-4df8-8dfa-7bceb6238555\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") "
Oct 31 04:22:31.162848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.162661    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") pod \"316c7c51-4157-4df8-8dfa-7bceb6238555\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") "
Oct 31 04:22:31.162848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.162682    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\" (UniqueName: \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\") pod \"316c7c51-4157-4df8-8dfa-7bceb6238555\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") "
Oct 31 04:22:31.162848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.162823    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\") pod \"316c7c51-4157-4df8-8dfa-7bceb6238555\" (UID: \"316c7c51-4157-4df8-8dfa-7bceb6238555\") "
Oct 31 04:22:31.163124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.163100    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca" pod "316c7c51-4157-4df8-8dfa-7bceb6238555" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:31.163382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.163364    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316c7c51-4157-4df8-8dfa-7bceb6238555-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:31.164914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.164890    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls" pod "316c7c51-4157-4df8-8dfa-7bceb6238555" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:31.164985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.164914    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs" pod "316c7c51-4157-4df8-8dfa-7bceb6238555" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:31.165024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.164999    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m" pod "316c7c51-4157-4df8-8dfa-7bceb6238555" (UID: "316c7c51-4157-4df8-8dfa-7bceb6238555"). InnerVolumeSpecName "kube-api-access-6f57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:31.266956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.266833    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:31.266956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.266876    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f57m\" (UniqueName: \"kubernetes.io/projected/316c7c51-4157-4df8-8dfa-7bceb6238555-kube-api-access-6f57m\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:31.266956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.266893    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316c7c51-4157-4df8-8dfa-7bceb6238555-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:31.336422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.336383    6931 generic.go:356] "Generic (PLEG): container finished" podID="316c7c51-4157-4df8-8dfa-7bceb6238555" containerID="0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe" exitCode=0
Oct 31 04:22:31.336621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.336436    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"
Oct 31 04:22:31.336621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.336447    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" event={"ID":"316c7c51-4157-4df8-8dfa-7bceb6238555","Type":"ContainerDied","Data":"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"}
Oct 31 04:22:31.336621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.336499    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz" event={"ID":"316c7c51-4157-4df8-8dfa-7bceb6238555","Type":"ContainerDied","Data":"818a55fa4bcdafb48ee7948d6c4fe0aa426159a469cc731dcafff20a38467577"}
Oct 31 04:22:31.336621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.336527    6931 scope.go:122] "RemoveContainer" containerID="0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"
Oct 31 04:22:31.348456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.348435    6931 scope.go:122] "RemoveContainer" containerID="0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"
Oct 31 04:22:31.348753 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:22:31.348733    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe\": container with ID starting with 0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe not found: ID does not exist" containerID="0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"
Oct 31 04:22:31.348812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.348761    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe"} err="failed to get container status \"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe\": rpc error: code = NotFound desc = could not find container \"0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe\": container with ID starting with 0c8349a25ae7f511f9eee66830dd712584e8dd3c59b9379e276f6f29fa157ffe not found: ID does not exist"
Oct 31 04:22:31.387240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.387210    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:22:31.399051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.399021    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8c0086-77c67cb4cd-dj4sz"]
Oct 31 04:22:31.673704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:31.673671    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:32.122734 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:32.122700    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316c7c51-4157-4df8-8dfa-7bceb6238555" path="/var/lib/kubelet/pods/316c7c51-4157-4df8-8dfa-7bceb6238555/volumes"
Oct 31 04:22:32.827492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:32.827458    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:33.113818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:33.113718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:33.704304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:33.704251    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:33.924382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:33.924345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:34.225823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:34.225791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:35.039234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:35.039201    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:35.402867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:35.402782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:36.495408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:36.495373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:38.302891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:38.302851    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:40.079467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:40.079434    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:40.605081 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:40.605004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:40.763658 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:22:40.763616    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:22:40.825702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:40.825666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:41.774287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:41.774257    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:42.286572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:42.286535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:44.396093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:44.396060    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:45.416514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:45.416476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:45.620593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:45.620530    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:46.243729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:46.243685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:46.456989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:46.456955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:46.737763 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:46.737405    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:47.530428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:47.530389    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:47.863564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:47.863474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:48.415396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:48.415359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:49.570040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:49.569999    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:49.783144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:49.783109    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:50.201481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:50.201437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:51.550291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.550246    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:22:51.551095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.551042    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" podUID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" containerName="maas-api" containerID="cri-o://d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553" gracePeriod=30
Oct 31 04:22:51.718015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.717969    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:51.910451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.910418    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:22:51.978295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.978251    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\") pod \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") "
Oct 31 04:22:51.978483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.978444    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\" (UniqueName: \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\") pod \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") "
Oct 31 04:22:51.978533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.978517    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\") pod \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") "
Oct 31 04:22:51.978707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.978689    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\") pod \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\" (UID: \"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99\") "
Oct 31 04:22:51.980594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.980565    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca" pod "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" (UID: "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:51.986433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.985602    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf" pod "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" (UID: "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99"). InnerVolumeSpecName "kube-api-access-mx4rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:51.995504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.995449    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs" pod "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" (UID: "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:51.997650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:51.997591    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls" pod "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" (UID: "efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:52.080651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.080617    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:52.080651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.080646    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4rf\" (UniqueName: \"kubernetes.io/projected/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-kube-api-access-mx4rf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:52.080651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.080657    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:52.080885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.080666    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:52.156473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.156437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:52.366733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.366701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:52.508921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.508830    6931 generic.go:356] "Generic (PLEG): container finished" podID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" containerID="d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553" exitCode=0
Oct 31 04:22:52.509324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.509301    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" event={"ID":"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99","Type":"ContainerDied","Data":"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"}
Oct 31 04:22:52.509490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.509472    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x" event={"ID":"efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99","Type":"ContainerDied","Data":"b0b29223ae2e3a9afcf242420c8ea3d857d1c0538481648b352e61a33af6812a"}
Oct 31 04:22:52.509609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.509597    6931 scope.go:122] "RemoveContainer" containerID="d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"
Oct 31 04:22:52.509868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.509852    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"
Oct 31 04:22:52.527232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.527196    6931 scope.go:122] "RemoveContainer" containerID="d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"
Oct 31 04:22:52.528964 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:22:52.528938    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553\": container with ID starting with d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553 not found: ID does not exist" containerID="d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"
Oct 31 04:22:52.529137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.529112    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553"} err="failed to get container status \"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553\": rpc error: code = NotFound desc = could not find container \"d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553\": container with ID starting with d02a5250091b39b413069311d330e899a16559ae663216c380f2c270aaf03553 not found: ID does not exist"
Oct 31 04:22:52.569838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.569804    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:22:52.574534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.574504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:52.581334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:52.581310    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-t8m4x"]
Oct 31 04:22:53.132616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:53.132584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:54.129569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:54.129536    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" path="/var/lib/kubelet/pods/efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99/volumes"
Oct 31 04:22:54.157740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:54.157679    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:54.365580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:54.365540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:54.589859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:54.589823    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:54.861412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:54.861320    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.083214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:55.083183    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.513377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:55.513339    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.744472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:55.744437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:56.169622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:56.169587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:56.627342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:56.627305    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:56.857674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:56.857564    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:57.123619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.123413    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:57.345837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.345804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:57.441466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.441371    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:22:57.441656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.441631    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" podUID="0d129d68-c6f1-42dd-af60-11c02795999e" containerName="maas-api" containerID="cri-o://e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a" gracePeriod=30
Oct 31 04:22:57.559059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.559026    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:57.762381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.762343    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:22:57.778909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.778871    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:57.865257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.865222    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") pod \"0d129d68-c6f1-42dd-af60-11c02795999e\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") "
Oct 31 04:22:57.865453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.865336    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\") pod \"0d129d68-c6f1-42dd-af60-11c02795999e\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") "
Oct 31 04:22:57.865453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.865364    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\") pod \"0d129d68-c6f1-42dd-af60-11c02795999e\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") "
Oct 31 04:22:57.865574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.865481    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\" (UniqueName: \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\") pod \"0d129d68-c6f1-42dd-af60-11c02795999e\" (UID: \"0d129d68-c6f1-42dd-af60-11c02795999e\") "
Oct 31 04:22:57.865895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.865868    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca" pod "0d129d68-c6f1-42dd-af60-11c02795999e" (UID: "0d129d68-c6f1-42dd-af60-11c02795999e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:57.866098 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.866070    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d129d68-c6f1-42dd-af60-11c02795999e-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:57.868107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.867715    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7" pod "0d129d68-c6f1-42dd-af60-11c02795999e" (UID: "0d129d68-c6f1-42dd-af60-11c02795999e"). InnerVolumeSpecName "kube-api-access-r6vp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:57.868307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.868287    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs" pod "0d129d68-c6f1-42dd-af60-11c02795999e" (UID: "0d129d68-c6f1-42dd-af60-11c02795999e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:57.868382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.868326    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls" pod "0d129d68-c6f1-42dd-af60-11c02795999e" (UID: "0d129d68-c6f1-42dd-af60-11c02795999e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:57.967748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.967709    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:57.967748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.967742    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vp7\" (UniqueName: \"kubernetes.io/projected/0d129d68-c6f1-42dd-af60-11c02795999e-kube-api-access-r6vp7\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:57.967748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.967752    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d129d68-c6f1-42dd-af60-11c02795999e-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:22:57.990091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:57.985640    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:58.197506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.197474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:58.407496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.407458    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:58.558635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.558601    6931 generic.go:356] "Generic (PLEG): container finished" podID="0d129d68-c6f1-42dd-af60-11c02795999e" containerID="e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a" exitCode=0
Oct 31 04:22:58.558837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.558650    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"
Oct 31 04:22:58.558837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.558690    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" event={"ID":"0d129d68-c6f1-42dd-af60-11c02795999e","Type":"ContainerDied","Data":"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"}
Oct 31 04:22:58.558837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.558741    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm" event={"ID":"0d129d68-c6f1-42dd-af60-11c02795999e","Type":"ContainerDied","Data":"4f5c6c75e2ddba3792ec2f2f73326c9c4b246f006ca212a24cb5fbe6cf0cce28"}
Oct 31 04:22:58.558837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.558764    6931 scope.go:122] "RemoveContainer" containerID="e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"
Oct 31 04:22:58.568776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.568756    6931 scope.go:122] "RemoveContainer" containerID="e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"
Oct 31 04:22:58.569087 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:22:58.569050    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a\": container with ID starting with e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a not found: ID does not exist" containerID="e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"
Oct 31 04:22:58.569147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.569093    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a"} err="failed to get container status \"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a\": rpc error: code = NotFound desc = could not find container \"e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a\": container with ID starting with e1bc885611412e32d5a01a64f1037c0f4ca093a1e23d2df60cc8e18cc5f9705a not found: ID does not exist"
Oct 31 04:22:58.616891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.616856    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:22:58.628342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:22:58.628299    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a43a70-846dc884c6-vsthm"]
Oct 31 04:23:00.138746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:00.138280    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d129d68-c6f1-42dd-af60-11c02795999e" path="/var/lib/kubelet/pods/0d129d68-c6f1-42dd-af60-11c02795999e/volumes"
Oct 31 04:23:00.726994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:00.726955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:00.932582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:00.932535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:01.435175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:01.435104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:01.670788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:01.670755    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:01.880150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:01.880051    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:02.095705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:02.095671    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:02.521413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:02.521373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:02.733234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:02.733201    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:02.960678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:02.960596    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:03.451241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:03.451204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:03.885904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:03.885861    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:04.117636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:04.117595    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:04.403233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:04.403190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:04.880900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:04.880862    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:05.090575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:05.090540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:05.689479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:05.689448    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:05.937598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:05.937563    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:07.669624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.669577    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:23:07.670898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.670872    6931 cpu_manager.go:414] "containerMap: removing container" podUID="316c7c51-4157-4df8-8dfa-7bceb6238555" containerName="maas-api"
Oct 31 04:23:07.670898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.670899    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316c7c51-4157-4df8-8dfa-7bceb6238555" containerName="maas-api"
Oct 31 04:23:07.671083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.670983    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0d129d68-c6f1-42dd-af60-11c02795999e" containerName="maas-api"
Oct 31 04:23:07.671083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.670992    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d129d68-c6f1-42dd-af60-11c02795999e" containerName="maas-api"
Oct 31 04:23:07.671083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.671030    6931 cpu_manager.go:414] "containerMap: removing container" podUID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" containerName="maas-api"
Oct 31 04:23:07.671083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.671038    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" containerName="maas-api"
Oct 31 04:23:07.671346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.671330    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d129d68-c6f1-42dd-af60-11c02795999e" containerName="maas-api"
Oct 31 04:23:07.671397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.671357    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="316c7c51-4157-4df8-8dfa-7bceb6238555" containerName="maas-api"
Oct 31 04:23:07.671397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.671376    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdf93d6-0c51-45e6-8a7b-cbc0e9d32e99" containerName="maas-api"
Oct 31 04:23:07.678919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.678894    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.691597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.691564    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:23:07.700463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.700436    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-146320d5-serving-cert\""
Oct 31 04:23:07.805935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.805892    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.806131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.805946    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntq4\" (UniqueName: \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.806131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.806073    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.806297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.806240    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.872777 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.872701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:07.907856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.907815    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.908058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.907875    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntq4\" (UniqueName: \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.908058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.907940    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.908058 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:23:07.907973    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-serving-cert: secret "maas-api-e2e-mt-146320d5-serving-cert" not found
Oct 31 04:23:07.908058 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:23:07.908053    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls podName:a892c415-91dc-4728-ac31-9e6b78b5b2e8 nodeName:}" failed. No retries permitted until 2026-10-31 04:23:08.408037641 +0000 UTC m=+1084.441037297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls") pod "maas-api-e2e-mt-146320d5-8478684b9-7bddq" (UID: "a892c415-91dc-4728-ac31-9e6b78b5b2e8") : secret "maas-api-e2e-mt-146320d5-serving-cert" not found
Oct 31 04:23:07.908385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.908093    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.908722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.908700    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.911099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.911073    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:07.932375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:07.932295    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntq4\" (UniqueName: \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:08.089914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.089869    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:08.299981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.299946    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:08.415944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.415905    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:08.418368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.418344    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") pod \"maas-api-e2e-mt-146320d5-8478684b9-7bddq\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:08.597399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.597243    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:08.919335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:08.919244    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:23:08.943499 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:23:08.943453    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda892c415_91dc_4728_ac31_9e6b78b5b2e8.slice/crio-7b63413a3feea5f98d130c07aa38bab98af48c4fb5d863d6bf05cc23edc4be0e WatchSource:0}: Error finding container 7b63413a3feea5f98d130c07aa38bab98af48c4fb5d863d6bf05cc23edc4be0e: Status 404 returned error can't find the container with id 7b63413a3feea5f98d130c07aa38bab98af48c4fb5d863d6bf05cc23edc4be0e
Oct 31 04:23:09.311636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.311600    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:09.539611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.539571    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:09.634427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.634339    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" event={"ID":"a892c415-91dc-4728-ac31-9e6b78b5b2e8","Type":"ContainerStarted","Data":"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"}
Oct 31 04:23:09.634427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.634397    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" event={"ID":"a892c415-91dc-4728-ac31-9e6b78b5b2e8","Type":"ContainerStarted","Data":"7b63413a3feea5f98d130c07aa38bab98af48c4fb5d863d6bf05cc23edc4be0e"}
Oct 31 04:23:09.634671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.634480    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:09.675987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:09.675935    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" podStartSLOduration=2.290305852 podStartE2EDuration="2.675920697s" totalImagesPullingTime="385.614845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:09.674829281 +0000 UTC m=+1085.707828990" watchObservedRunningTime="2026-10-31 04:23:09.675920697 +0000 UTC m=+1085.708920376"
Oct 31 04:23:10.145352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:10.145310    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:10.760127 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:23:10.760093    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:23:11.004575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:11.004529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:11.228182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:11.227802    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:12.114890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:12.114844    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:13.996263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:13.996222    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:15.646571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:15.646543    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:23:15.846206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:15.846147    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:16.055196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:16.055135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:17.848387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:17.847673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:19.486679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.486644    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:23:19.491009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.490993    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.528232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.528196    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:23:19.532810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.532783    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f91cd1-serving-cert\""
Oct 31 04:23:19.569743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.569707    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.569885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.569772    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.569885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.569867    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.570005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.569988    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68d2q\" (UniqueName: \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.671131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.671091    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68d2q\" (UniqueName: \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.671338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.671299    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.671381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.671339    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.671381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.671370    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.672019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.671991    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.673860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.673841    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.673938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.673902    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.684259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.684203    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:19.708657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.708616    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68d2q\" (UniqueName: \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\") pod \"maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.805113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.805074    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:19.904203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:19.904132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:20.120121 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:23:20.120082    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bab208_0bae_4a6c_a50d_7fbe8c5825e8.slice/crio-7e17f15a816550655a76b0215d3e39c8321852a7f7ad124c87b9a9f4559c04e2 WatchSource:0}: Error finding container 7e17f15a816550655a76b0215d3e39c8321852a7f7ad124c87b9a9f4559c04e2: Status 404 returned error can't find the container with id 7e17f15a816550655a76b0215d3e39c8321852a7f7ad124c87b9a9f4559c04e2
Oct 31 04:23:20.124466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:20.124447    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:23:20.733251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:20.733125    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" event={"ID":"10bab208-0bae-4a6c-a50d-7fbe8c5825e8","Type":"ContainerStarted","Data":"2540c5f7d767bb608e2521736aa3337795cb85e3e02df520eed83df76a569914"}
Oct 31 04:23:20.733746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:20.733265    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" event={"ID":"10bab208-0bae-4a6c-a50d-7fbe8c5825e8","Type":"ContainerStarted","Data":"7e17f15a816550655a76b0215d3e39c8321852a7f7ad124c87b9a9f4559c04e2"}
Oct 31 04:23:20.733746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:20.733530    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:20.805930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:20.805827    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" podStartSLOduration=1.491365167 podStartE2EDuration="1.805805251s" totalImagesPullingTime="314.440084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:20.803589103 +0000 UTC m=+1096.836588782" watchObservedRunningTime="2026-10-31 04:23:20.805805251 +0000 UTC m=+1096.838804931"
Oct 31 04:23:21.726204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:21.726145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:23.643083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:23.643050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:25.259765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.259726    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:23:25.264331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.264309    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.281979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.281952    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ceba51-serving-cert\""
Oct 31 04:23:25.299040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.299000    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:23:25.365100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.365066    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.365100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.365110    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.365343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.365220    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2c6\" (UniqueName: \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.365395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.365380    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.459894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.459850    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:25.466347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.466324    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.466444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.466358    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.466444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.466387    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2c6\" (UniqueName: \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.466550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.466507    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.467194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.467171    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.469057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.469031    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.473514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.473464    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.499314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.495512    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2c6\" (UniqueName: \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\") pod \"maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.579322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.579276    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:25.676133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.676091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:25.886585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:25.886508    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:23:25.889986 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:23:25.889957    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0e0f23_da23_458a_8afe_4a82c72a8a6b.slice/crio-85b7aacabb9a222ee7829348e30acb5544f8a2b63a5a0f2571133fc4df4c6941 WatchSource:0}: Error finding container 85b7aacabb9a222ee7829348e30acb5544f8a2b63a5a0f2571133fc4df4c6941: Status 404 returned error can't find the container with id 85b7aacabb9a222ee7829348e30acb5544f8a2b63a5a0f2571133fc4df4c6941
Oct 31 04:23:26.746444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:26.746373    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:23:26.775254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:26.775219    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" event={"ID":"8e0e0f23-da23-458a-8afe-4a82c72a8a6b","Type":"ContainerStarted","Data":"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"}
Oct 31 04:23:26.775414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:26.775272    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" event={"ID":"8e0e0f23-da23-458a-8afe-4a82c72a8a6b","Type":"ContainerStarted","Data":"85b7aacabb9a222ee7829348e30acb5544f8a2b63a5a0f2571133fc4df4c6941"}
Oct 31 04:23:26.775414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:26.775302    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:26.833567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:26.833004    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" podStartSLOduration=1.40839853 podStartE2EDuration="1.832980238s" totalImagesPullingTime="424.581708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:26.829411891 +0000 UTC m=+1102.862411571" watchObservedRunningTime="2026-10-31 04:23:26.832980238 +0000 UTC m=+1102.865979920"
Oct 31 04:23:27.486181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:27.482294    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:29.484835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:29.484785    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:31.435834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:31.435794    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:31.647776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:31.647723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:31.878440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:31.878398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:32.351026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:32.350987    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:32.571508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:32.571476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:32.788851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:32.788820    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:23:33.507467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:33.507389    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:35.458876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:35.458840    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:36.118097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:36.118057    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:36.333324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:36.333290    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:37.583015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:37.582976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:38.957571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:38.957532    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:39.188263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:39.188227    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:39.398437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:39.398399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:39.830216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:39.830141    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:40.759376 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:23:40.759338    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:23:41.757113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:41.757068    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:42.322578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:42.322536    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:42.534958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:42.534908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:43.454484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:43.454442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:44.167544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:44.167504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:44.383840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:44.383803    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:44.815576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:44.815538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:45.031137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:45.031099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:46.274979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:46.274943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:48.887017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:48.886975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:49.116979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:49.116948    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:50.143198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:50.143166    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:51.904133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:51.904099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:53.506452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:53.506412    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:55.399632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:55.399593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:56.930380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:56.930336    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:57.169857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.169817    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:23:57.173990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.173962    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.224117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.224067    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-801218-serving-cert\""
Oct 31 04:23:57.240787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.240750    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:23:57.241425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.241398    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.242013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.241987    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48p2\" (UniqueName: \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.242108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.242052    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.242108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.242080    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.343233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.343200    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48p2\" (UniqueName: \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.343413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.343253    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.343413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.343280    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.343413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.343329    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.343967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.343943    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.345828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.345805    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.346028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.346001    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.350319 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.350287    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:57.368855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.368834    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48p2\" (UniqueName: \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\") pod \"maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.497614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.497376    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:57.594036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.593993    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:57.815251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:57.815220    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:23:57.818933 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:23:57.818905    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79821c1b_2b14_46c5_aa07_c8e62ce5a355.slice/crio-5a0d3ddd8627556ce0d3ff19eee5376355bcc3e969dc7f69fa02c93692de30bd WatchSource:0}: Error finding container 5a0d3ddd8627556ce0d3ff19eee5376355bcc3e969dc7f69fa02c93692de30bd: Status 404 returned error can't find the container with id 5a0d3ddd8627556ce0d3ff19eee5376355bcc3e969dc7f69fa02c93692de30bd
Oct 31 04:23:58.028592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:58.028553    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" event={"ID":"79821c1b-2b14-46c5-aa07-c8e62ce5a355","Type":"ContainerStarted","Data":"5a0d3ddd8627556ce0d3ff19eee5376355bcc3e969dc7f69fa02c93692de30bd"}
Oct 31 04:23:59.039073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:59.039036    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" event={"ID":"79821c1b-2b14-46c5-aa07-c8e62ce5a355","Type":"ContainerStarted","Data":"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"}
Oct 31 04:23:59.039621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:59.039103    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:23:59.085513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:59.085458    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" podStartSLOduration=1.712067276 podStartE2EDuration="2.085445021s" totalImagesPullingTime="373.377745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:59.082376959 +0000 UTC m=+1135.115376671" watchObservedRunningTime="2026-10-31 04:23:59.085445021 +0000 UTC m=+1135.118444701"
Oct 31 04:23:59.750650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:23:59.750612    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:01.224525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.224489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:01.447108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.447056    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:01.884185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.884110    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:24:01.888752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.888728    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:01.910787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.910749    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:24:01.915566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.915527    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3430f6-dockercfg-bxjgk\""
Oct 31 04:24:01.923638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:01.923600    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3430f6\""
Oct 31 04:24:02.011858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.011822    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.012064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.012044    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9rg\" (UniqueName: \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.113612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.113579    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9rg\" (UniqueName: \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.113796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.113704    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.114413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.114395    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.136999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.136936    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9rg\" (UniqueName: \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\") pod \"payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.206198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.206141    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:02.513998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.513965    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:24:02.517200 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:24:02.517170    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63510c2c_b057_46d3_ad5c_20c11a1ba97c.slice/crio-e78e12a8b806c6b8d9cad906099b1074935b1156b4d46fc1d713b40a8425e8f7 WatchSource:0}: Error finding container e78e12a8b806c6b8d9cad906099b1074935b1156b4d46fc1d713b40a8425e8f7: Status 404 returned error can't find the container with id e78e12a8b806c6b8d9cad906099b1074935b1156b4d46fc1d713b40a8425e8f7
Oct 31 04:24:02.880651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:02.880612    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:03.066008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:03.065973    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" event={"ID":"63510c2c-b057-46d3-ad5c-20c11a1ba97c","Type":"ContainerStarted","Data":"e78e12a8b806c6b8d9cad906099b1074935b1156b4d46fc1d713b40a8425e8f7"}
Oct 31 04:24:04.075805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:04.075724    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" event={"ID":"63510c2c-b057-46d3-ad5c-20c11a1ba97c","Type":"ContainerStarted","Data":"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"}
Oct 31 04:24:04.076173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:04.075807    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:04.117058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:04.116996    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" podStartSLOduration=1.826706269 podStartE2EDuration="3.116978154s" totalImagesPullingTime="1.290271885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:04.11545396 +0000 UTC m=+1140.148453662" watchObservedRunningTime="2026-10-31 04:24:04.116978154 +0000 UTC m=+1140.149977835"
Oct 31 04:24:04.895055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:04.895009    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:05.052113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:05.052087    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:24:06.311684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.311639    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:06.517594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.517553    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:06.759951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.759872    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:24:06.764737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.764709    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.781674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.781292    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.781674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.781598    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxbd\" (UniqueName: \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.791343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.791317    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f91cd1-dockercfg-lzxfr\""
Oct 31 04:24:06.796728 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.794816    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:24:06.796897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.796672    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f91cd1\""
Oct 31 04:24:06.883205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.883169    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxbd\" (UniqueName: \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.883397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.883223    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.883889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.883869    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:06.905740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:06.905708    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxbd\" (UniqueName: \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\") pod \"payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:07.082192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:07.082147    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:07.385696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:07.385667    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:24:07.388802 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:24:07.388774    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef411f3_5700_4185_821b_6b265c9ba643.slice/crio-a0f9aac2317c376df50e1adea95b987988d3af487dabe87f69c4f6d3607cf3b6 WatchSource:0}: Error finding container a0f9aac2317c376df50e1adea95b987988d3af487dabe87f69c4f6d3607cf3b6: Status 404 returned error can't find the container with id a0f9aac2317c376df50e1adea95b987988d3af487dabe87f69c4f6d3607cf3b6
Oct 31 04:24:07.666491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:07.666398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:08.104993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:08.104959    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" event={"ID":"0ef411f3-5700-4185-821b-6b265c9ba643","Type":"ContainerStarted","Data":"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"}
Oct 31 04:24:08.105199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:08.105015    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" event={"ID":"0ef411f3-5700-4185-821b-6b265c9ba643","Type":"ContainerStarted","Data":"a0f9aac2317c376df50e1adea95b987988d3af487dabe87f69c4f6d3607cf3b6"}
Oct 31 04:24:08.105199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:08.105180    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:08.148186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:08.148098    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" podStartSLOduration=2.148079052 podStartE2EDuration="2.148079052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:08.145926708 +0000 UTC m=+1144.178926391" watchObservedRunningTime="2026-10-31 04:24:08.148079052 +0000 UTC m=+1144.181078733"
Oct 31 04:24:09.517495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:09.517461    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:10.870079 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:24:10.862368    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:24:11.759448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:11.759409    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:13.190814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:13.190774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:13.407102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:13.407059    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:14.689181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:14.689111    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:14.908421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:14.908357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:15.084317 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:15.084287    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:24:17.234726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:17.234689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:19.036764 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:19.036730    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:19.114071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:19.114042    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:24:19.328278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:19.328193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:19.810181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:19.802561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:20.321426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:20.321388    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:20.543131 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:20.543095    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:21.092542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:21.092505    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:21.309714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:21.309678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:21.799641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:21.799600    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:23.335777 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:23.335739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:23.770296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:23.770198    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:23.995011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:23.994984    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:25.487262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:25.487225    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:25.702513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:25.702470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:27.829365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:27.829312    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:28.078787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:28.078754    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:30.576863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:30.576811    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:32.212610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:32.212575    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:32.426248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:32.426208    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:32.644808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:32.644770    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:32.858815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:32.858774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:33.068675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:33.068637    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:33.424460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:33.424377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:33.632753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:33.632708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:35.160309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:35.160268    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:37.005817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:37.005784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:37.212147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:37.212102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:37.547973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:37.547938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:37.758300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:37.758258    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:38.742299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:38.742245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:39.025247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:39.023116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:40.170593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:40.170550    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:40.533569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:40.533531    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:40.761853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:40.761765    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:40.831899 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:24:40.831818    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:24:42.971549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:42.971451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:44.178685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.178606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:44.390046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.390012    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:44.419621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.419586    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:24:44.419872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.419826    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" podUID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" containerName="maas-api" containerID="cri-o://f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc" gracePeriod=30
Oct 31 04:24:44.597351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.597315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:44.771239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.771199    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:24:44.938951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.938851    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") pod \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") "
Oct 31 04:24:44.938951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.938941    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\" (UniqueName: \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\") pod \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") "
Oct 31 04:24:44.939215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.939098    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\") pod \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") "
Oct 31 04:24:44.939276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.939215    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\") pod \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\" (UID: \"a892c415-91dc-4728-ac31-9e6b78b5b2e8\") "
Oct 31 04:24:44.939884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.939653    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca" pod "a892c415-91dc-4728-ac31-9e6b78b5b2e8" (UID: "a892c415-91dc-4728-ac31-9e6b78b5b2e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:24:44.939884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.939851    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a892c415-91dc-4728-ac31-9e6b78b5b2e8-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:24:44.943491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.943463    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls" pod "a892c415-91dc-4728-ac31-9e6b78b5b2e8" (UID: "a892c415-91dc-4728-ac31-9e6b78b5b2e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:24:44.946575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.946551    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4" pod "a892c415-91dc-4728-ac31-9e6b78b5b2e8" (UID: "a892c415-91dc-4728-ac31-9e6b78b5b2e8"). InnerVolumeSpecName "kube-api-access-qntq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:24:44.946759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:44.946662    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs" pod "a892c415-91dc-4728-ac31-9e6b78b5b2e8" (UID: "a892c415-91dc-4728-ac31-9e6b78b5b2e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:24:45.040798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.040766    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:24:45.040798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.040795    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a892c415-91dc-4728-ac31-9e6b78b5b2e8-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:24:45.040798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.040805    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntq4\" (UniqueName: \"kubernetes.io/projected/a892c415-91dc-4728-ac31-9e6b78b5b2e8-kube-api-access-qntq4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:24:45.388172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.388125    6931 generic.go:356] "Generic (PLEG): container finished" podID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" containerID="f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc" exitCode=0
Oct 31 04:24:45.388580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.388185    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"
Oct 31 04:24:45.388580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.388214    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" event={"ID":"a892c415-91dc-4728-ac31-9e6b78b5b2e8","Type":"ContainerDied","Data":"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"}
Oct 31 04:24:45.388580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.388275    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq" event={"ID":"a892c415-91dc-4728-ac31-9e6b78b5b2e8","Type":"ContainerDied","Data":"7b63413a3feea5f98d130c07aa38bab98af48c4fb5d863d6bf05cc23edc4be0e"}
Oct 31 04:24:45.388580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.388300    6931 scope.go:122] "RemoveContainer" containerID="f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"
Oct 31 04:24:45.398420 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.398403    6931 scope.go:122] "RemoveContainer" containerID="f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"
Oct 31 04:24:45.398732 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:24:45.398708    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc\": container with ID starting with f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc not found: ID does not exist" containerID="f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"
Oct 31 04:24:45.398800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.398745    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc"} err="failed to get container status \"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc\": rpc error: code = NotFound desc = could not find container \"f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc\": container with ID starting with f7417e57821d58b87a31e948d25fea1db727f5e1ae67498888c8ce62255149dc not found: ID does not exist"
Oct 31 04:24:45.440992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.440958    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:24:45.452635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.452596    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-146320d5-8478684b9-7bddq"]
Oct 31 04:24:45.805963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:45.805930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:46.123270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:46.123181    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" path="/var/lib/kubelet/pods/a892c415-91dc-4728-ac31-9e6b78b5b2e8/volumes"
Oct 31 04:24:47.703372 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:47.703326    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:49.357268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.357220    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:24:49.358057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.358037    6931 cpu_manager.go:414] "containerMap: removing container" podUID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" containerName="maas-api"
Oct 31 04:24:49.358188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.358060    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" containerName="maas-api"
Oct 31 04:24:49.358282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.358260    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a892c415-91dc-4728-ac31-9e6b78b5b2e8" containerName="maas-api"
Oct 31 04:24:49.363017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.362999    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.427342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.427300    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.505302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.505270    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.505844 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.505820    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.506073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.506054    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nj4j\" (UniqueName: \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.506517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.431938    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7b4185-serving-cert\""
Oct 31 04:24:49.517021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.516981    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:24:49.560782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.560746    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:49.607909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.607817    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.607909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.607899    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nj4j\" (UniqueName: \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.608191 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:24:49.607974    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-serving-cert: secret "maas-api-e2e-ipp-b-7b4185-serving-cert" not found
Oct 31 04:24:49.608191 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:24:49.608062    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls podName:e6da865f-eb7f-4419-a6f0-8642fa00221b nodeName:}" failed. No retries permitted until 2026-10-31 04:24:50.108042182 +0000 UTC m=+1186.141041853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls") pod "maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" (UID: "e6da865f-eb7f-4419-a6f0-8642fa00221b") : secret "maas-api-e2e-ipp-b-7b4185-serving-cert" not found
Oct 31 04:24:49.608320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.608201    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.608320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.608283    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.608936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.608912    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.611287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.611265    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.633449 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.633420    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nj4j\" (UniqueName: \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:49.773681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.773643    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:49.993273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:49.993181    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:50.120238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.117983    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:50.121559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.121526    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b4185-b684859db-8m84w\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:50.278340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.278175    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:50.280564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.279999    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:50.496879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.496839    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:50.600112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:50.600076    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:24:50.604944 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:24:50.604902    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6da865f_eb7f_4419_a6f0_8642fa00221b.slice/crio-9ff583ee6e8438f5b4e99af740657885b1f3b2287db7debc4d498e5db7450ac1 WatchSource:0}: Error finding container 9ff583ee6e8438f5b4e99af740657885b1f3b2287db7debc4d498e5db7450ac1: Status 404 returned error can't find the container with id 9ff583ee6e8438f5b4e99af740657885b1f3b2287db7debc4d498e5db7450ac1
Oct 31 04:24:51.521698 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:51.521652    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" event={"ID":"e6da865f-eb7f-4419-a6f0-8642fa00221b","Type":"ContainerStarted","Data":"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"}
Oct 31 04:24:51.522097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:51.521748    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" event={"ID":"e6da865f-eb7f-4419-a6f0-8642fa00221b","Type":"ContainerStarted","Data":"9ff583ee6e8438f5b4e99af740657885b1f3b2287db7debc4d498e5db7450ac1"}
Oct 31 04:24:51.522097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:51.521773    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:51.565538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:51.565466    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" podStartSLOduration=2.213861945 podStartE2EDuration="2.565450047s" totalImagesPullingTime="351.588102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:51.565438776 +0000 UTC m=+1187.598438454" watchObservedRunningTime="2026-10-31 04:24:51.565450047 +0000 UTC m=+1187.598449727"
Oct 31 04:24:52.180038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.180004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:52.656792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.656761    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:24:52.661667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.661643    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.677930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.677905    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f91cd1\"/\"openshift-service-ca.crt\""
Oct 31 04:24:52.690056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.690024    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f91cd1\"/\"default-dockercfg-ld7bb\""
Oct 31 04:24:52.690394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.690374    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f91cd1\"/\"kube-root-ca.crt\""
Oct 31 04:24:52.695543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.695514    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:24:52.695685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.695668    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f91cd1\"/\"ipp-route-f91cd1-kserve-self-signed-certs\""
Oct 31 04:24:52.753232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753193    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47d4\" (UniqueName: \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.753431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753269    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.753511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753427    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.753556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753541    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.753614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753601    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.753663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.753645    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.805552 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.805514    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:52.854806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.854755    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.854865    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.854938    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855005    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855062    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855114    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47d4\" (UniqueName: \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855288    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855329    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.855390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.855362    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.857393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.857366    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.857937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.857916    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.878739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.878708    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47d4\" (UniqueName: \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\") pod \"ipp-route-f91cd1-kserve-858658d885-g9jjc\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") " pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:52.978513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:52.978421    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:53.154019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.153982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:53.301897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.301863    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:24:53.305416 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:24:53.305385    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8db1f49a_3c83_4899_b9cf_dc07701def71.slice/crio-462ec926bdf78fb48e35a0a205d87d2bff10c7f09148bfbafb64f3f8e485fcb0 WatchSource:0}: Error finding container 462ec926bdf78fb48e35a0a205d87d2bff10c7f09148bfbafb64f3f8e485fcb0: Status 404 returned error can't find the container with id 462ec926bdf78fb48e35a0a205d87d2bff10c7f09148bfbafb64f3f8e485fcb0
Oct 31 04:24:53.365992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.365962    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:53.538607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.538573    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" event={"ID":"8db1f49a-3c83-4899-b9cf-dc07701def71","Type":"ContainerStarted","Data":"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"}
Oct 31 04:24:53.538811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.538620    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" event={"ID":"8db1f49a-3c83-4899-b9cf-dc07701def71","Type":"ContainerStarted","Data":"462ec926bdf78fb48e35a0a205d87d2bff10c7f09148bfbafb64f3f8e485fcb0"}
Oct 31 04:24:53.580620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.580528    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" podStartSLOduration=1.580515965 podStartE2EDuration="1.580515965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:53.579831332 +0000 UTC m=+1189.612831034" watchObservedRunningTime="2026-10-31 04:24:53.580515965 +0000 UTC m=+1189.613515644"
Oct 31 04:24:53.979438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:53.979351    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:24:54.023181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:54.023133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:54.416669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:54.416636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:54.620136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:54.620092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:54.897858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:54.897826    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:55.125358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:55.125322    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:56.660676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:56.660638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:57.050720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:57.050690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:57.533295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:57.533270    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:24:59.178200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:24:59.178069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:01.113166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:01.113109    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:02.979754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:02.979719    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:25:02.992224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:02.992195    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:25:03.459482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:03.459446    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:03.624982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:03.624953    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:25:04.981457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:04.981421    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:25:04.986898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:04.986877    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.003607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.003579    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a9b573-serving-cert\""
Oct 31 04:25:05.022925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.022891    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:25:05.127603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.127563    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.127776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.127702    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.127776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.127752    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxn6w\" (UniqueName: \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.127937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.127923    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.178790 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.178725    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:05.229711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.229672    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.229911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.229746    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxn6w\" (UniqueName: \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.229911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.229905    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.230039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.229972    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.230188 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:05.230142    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-serving-cert: secret "maas-api-e2e-ipp-b-a9b573-serving-cert" not found
Oct 31 04:25:05.230353 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:05.230275    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls podName:416e232e-c795-4209-982a-08405291ede8 nodeName:}" failed. No retries permitted until 2026-10-31 04:25:05.730247656 +0000 UTC m=+1201.763247314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls") pod "maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" (UID: "416e232e-c795-4209-982a-08405291ede8") : secret "maas-api-e2e-ipp-b-a9b573-serving-cert" not found
Oct 31 04:25:05.230678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.230658    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.232517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.232468    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.255515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.255481    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxn6w\" (UniqueName: \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.391582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.391527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:05.618015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.617965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:05.740554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.740510    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.742958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.742931    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:05.832645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.832596    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:05.900464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:05.900378    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:06.050726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:06.050653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:06.206952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:06.206922    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:25:06.209286 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:25:06.209256    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416e232e_c795_4209_982a_08405291ede8.slice/crio-ac0623b7576e7aefd98b23f4cf1017570fa1e00488f3b5c614b4a2b939e36d4d WatchSource:0}: Error finding container ac0623b7576e7aefd98b23f4cf1017570fa1e00488f3b5c614b4a2b939e36d4d: Status 404 returned error can't find the container with id ac0623b7576e7aefd98b23f4cf1017570fa1e00488f3b5c614b4a2b939e36d4d
Oct 31 04:25:06.639403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:06.639368    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" event={"ID":"416e232e-c795-4209-982a-08405291ede8","Type":"ContainerStarted","Data":"ac0623b7576e7aefd98b23f4cf1017570fa1e00488f3b5c614b4a2b939e36d4d"}
Oct 31 04:25:07.562928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.562895    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:25:07.567138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.567120    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.601042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.601007    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9154a6-serving-cert\""
Oct 31 04:25:07.604830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.604800    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:25:07.647557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.647521    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" event={"ID":"416e232e-c795-4209-982a-08405291ede8","Type":"ContainerStarted","Data":"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"}
Oct 31 04:25:07.647736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.647606    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:07.670576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.670542    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.670728 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.670603    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.670816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.670804    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696x9\" (UniqueName: \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.670871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.670859    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.692540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.692477    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" podStartSLOduration=3.364193723 podStartE2EDuration="3.692459801s" totalImagesPullingTime="328.266078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:07.690980941 +0000 UTC m=+1203.723980621" watchObservedRunningTime="2026-10-31 04:25:07.692459801 +0000 UTC m=+1203.725459483"
Oct 31 04:25:07.764737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.764695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:07.772167 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.772133    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696x9\" (UniqueName: \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.772292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.772278    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.772352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.772313    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.772404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.772374    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.773002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.772975    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.774923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.774899    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.775035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.774903    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.797116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.797077    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696x9\" (UniqueName: \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\") pod \"maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.890334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.885611    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:07.983448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:07.983361    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:08.243631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:08.243608    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:25:08.246653 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:25:08.246625    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a111e24_61f3_4328_9fd8_2d9607435519.slice/crio-218d766cbb5174eba09a1b38eb56392041dbad165c9d680c81edfa86ea08efb1 WatchSource:0}: Error finding container 218d766cbb5174eba09a1b38eb56392041dbad165c9d680c81edfa86ea08efb1: Status 404 returned error can't find the container with id 218d766cbb5174eba09a1b38eb56392041dbad165c9d680c81edfa86ea08efb1
Oct 31 04:25:08.657004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:08.656962    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" event={"ID":"0a111e24-61f3-4328-9fd8-2d9607435519","Type":"ContainerStarted","Data":"218d766cbb5174eba09a1b38eb56392041dbad165c9d680c81edfa86ea08efb1"}
Oct 31 04:25:09.665125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:09.665086    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" event={"ID":"0a111e24-61f3-4328-9fd8-2d9607435519","Type":"ContainerStarted","Data":"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"}
Oct 31 04:25:09.665614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:09.665213    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:09.708883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:09.708821    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" podStartSLOduration=2.381561932 podStartE2EDuration="2.708807857s" totalImagesPullingTime="327.245925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:25:09.70554715 +0000 UTC m=+1205.738546841" watchObservedRunningTime="2026-10-31 04:25:09.708807857 +0000 UTC m=+1205.741807535"
Oct 31 04:25:09.807521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:09.807486    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:10.759489 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:10.759452    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:25:11.652204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:11.652132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:13.601956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:13.601920    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:13.662645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:13.662615    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:25:14.047755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:14.047718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:15.343404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:15.343368    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:15.565466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:15.565434    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:15.678229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:15.678113    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:25:16.714654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:16.714616    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:18.676031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:18.675994    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:20.594532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:20.594492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:22.649617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:22.649568    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:24.584862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:24.584821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:25.047707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:25.047668    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:25.267043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:25.267004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:26.600085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:26.600040    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:27.138708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:27.138665    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:28.910144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:28.910099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:30.788137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:30.788100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:33.253187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:33.253037    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:33.462572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:33.462535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:35.068652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:35.068610    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:36.891546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:36.891508    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:25:36.891973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:36.891720    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy" containerID="cri-o://097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2" gracePeriod=30
Oct 31 04:25:37.249777 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:37.249694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:38.403221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:38.403150    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:38.614314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:38.614267    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:39.483193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:39.483135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:40.759556 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:40.759519    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:25:41.632398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:41.632355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:42.173616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.173561    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:25:42.281464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281419    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281483    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281537    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281569    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281668    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281761    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281789    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.281890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281840    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.282079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.281903    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\") pod \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\" (UID: \"3f51be56-c3fd-4fe1-a020-cd45da9043d5\") "
Oct 31 04:25:42.282593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282194    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:42.282593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282213    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:42.282593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282419    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:42.282593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282452    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:42.282853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282788    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.282853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282811    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.282853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282834    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.282995 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.282875    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.283107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.283074    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:42.287481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.287454    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:25:42.287665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.287597    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "kube-api-access-mkpfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:42.288536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.288503    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:42.290422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.290399    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token" pod "3f51be56-c3fd-4fe1-a020-cd45da9043d5" (UID: "3f51be56-c3fd-4fe1-a020-cd45da9043d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:42.384621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.384577    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.384621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.384614    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.384621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.384626    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.384869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.384638    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f51be56-c3fd-4fe1-a020-cd45da9043d5-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.384869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.384651    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpfh\" (UniqueName: \"kubernetes.io/projected/3f51be56-c3fd-4fe1-a020-cd45da9043d5-kube-api-access-mkpfh\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:42.890826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.890740    6931 generic.go:356] "Generic (PLEG): container finished" podID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerID="097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2" exitCode=0
Oct 31 04:25:42.890826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.890801    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"
Oct 31 04:25:42.891004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.890825    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" event={"ID":"3f51be56-c3fd-4fe1-a020-cd45da9043d5","Type":"ContainerDied","Data":"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"}
Oct 31 04:25:42.891004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.890899    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m" event={"ID":"3f51be56-c3fd-4fe1-a020-cd45da9043d5","Type":"ContainerDied","Data":"182f03f969320674fb6d01d50f5316394cce1838b654333ae634b398d117b2f1"}
Oct 31 04:25:42.891004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.890922    6931 scope.go:122] "RemoveContainer" containerID="097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"
Oct 31 04:25:42.901738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.901714    6931 scope.go:122] "RemoveContainer" containerID="097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"
Oct 31 04:25:42.902036 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:42.902018    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2\": container with ID starting with 097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2 not found: ID does not exist" containerID="097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"
Oct 31 04:25:42.902081 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.902045    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2"} err="failed to get container status \"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2\": rpc error: code = NotFound desc = could not find container \"097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2\": container with ID starting with 097c13aeddbee250f38dee2dc066c709c04fa6902a3f8cdb16c97b40ac457da2 not found: ID does not exist"
Oct 31 04:25:42.958383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.958344    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:25:42.969528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:42.969495    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-jml8m"]
Oct 31 04:25:43.647873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:43.647834    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:44.125468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:44.125433    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" path="/var/lib/kubelet/pods/3f51be56-c3fd-4fe1-a020-cd45da9043d5/volumes"
Oct 31 04:25:44.162355 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:44.162314    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:45.019897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:45.019859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:46.602191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:46.602135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:48.391331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:48.391296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:50.246556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:50.246524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:50.473064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:50.473025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:50.688724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:50.688693    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:25:50.688910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:50.688884    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy" containerID="cri-o://250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b" gracePeriod=30
Oct 31 04:25:51.396537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:51.396499    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:51.889949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:51.889885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:52.108784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:52.108752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:53.542765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:53.542719    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:55.100171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.100080    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:25:55.100574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.100383    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" podUID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" containerName="maas-api" containerID="cri-o://a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69" gracePeriod=30
Oct 31 04:25:55.295183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.294762    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:55.427333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.427298    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:25:55.493634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.491457    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\" (UniqueName: \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\") pod \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") "
Oct 31 04:25:55.493634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.491612    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\") pod \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") "
Oct 31 04:25:55.493634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.491653    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\") pod \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") "
Oct 31 04:25:55.493634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.491856    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\") pod \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\" (UID: \"79821c1b-2b14-46c5-aa07-c8e62ce5a355\") "
Oct 31 04:25:55.493634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.492871    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca" pod "79821c1b-2b14-46c5-aa07-c8e62ce5a355" (UID: "79821c1b-2b14-46c5-aa07-c8e62ce5a355"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:55.511263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.511013    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls" pod "79821c1b-2b14-46c5-aa07-c8e62ce5a355" (UID: "79821c1b-2b14-46c5-aa07-c8e62ce5a355"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:25:55.511349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.511325    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2" pod "79821c1b-2b14-46c5-aa07-c8e62ce5a355" (UID: "79821c1b-2b14-46c5-aa07-c8e62ce5a355"). InnerVolumeSpecName "kube-api-access-t48p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:55.512519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.512485    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs" pod "79821c1b-2b14-46c5-aa07-c8e62ce5a355" (UID: "79821c1b-2b14-46c5-aa07-c8e62ce5a355"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:25:55.594043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.593909    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79821c1b-2b14-46c5-aa07-c8e62ce5a355-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:55.594043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.593960    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:55.594043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.593987    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79821c1b-2b14-46c5-aa07-c8e62ce5a355-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:55.594043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.594006    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48p2\" (UniqueName: \"kubernetes.io/projected/79821c1b-2b14-46c5-aa07-c8e62ce5a355-kube-api-access-t48p2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:55.969485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.969455    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:25:55.991093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.991049    6931 generic.go:356] "Generic (PLEG): container finished" podID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerID="250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b" exitCode=0
Oct 31 04:25:55.991312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.991096    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"
Oct 31 04:25:55.991312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.991130    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" event={"ID":"1a35242b-41d6-4821-912e-43b4ed6bb53f","Type":"ContainerDied","Data":"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"}
Oct 31 04:25:55.991312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.991213    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c" event={"ID":"1a35242b-41d6-4821-912e-43b4ed6bb53f","Type":"ContainerDied","Data":"5571adaefc54b55a35764085c3ff9425a67c1692b86f108ca2dee8978f9cee61"}
Oct 31 04:25:55.991312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.991240    6931 scope.go:122] "RemoveContainer" containerID="250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"
Oct 31 04:25:55.992884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.992861    6931 generic.go:356] "Generic (PLEG): container finished" podID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" containerID="a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69" exitCode=0
Oct 31 04:25:55.993004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.992918    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"
Oct 31 04:25:55.993004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.992949    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" event={"ID":"79821c1b-2b14-46c5-aa07-c8e62ce5a355","Type":"ContainerDied","Data":"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"}
Oct 31 04:25:55.993004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.992989    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh" event={"ID":"79821c1b-2b14-46c5-aa07-c8e62ce5a355","Type":"ContainerDied","Data":"5a0d3ddd8627556ce0d3ff19eee5376355bcc3e969dc7f69fa02c93692de30bd"}
Oct 31 04:25:55.998577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998553    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.998696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998606    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.998696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998666    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.998779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998708    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.998879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998859    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.998947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998928    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.999069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998943    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:55.999069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.998983    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.999069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999015    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:55.999069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999056    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.999349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999145    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\") pod \"1a35242b-41d6-4821-912e-43b4ed6bb53f\" (UID: \"1a35242b-41d6-4821-912e-43b4ed6bb53f\") "
Oct 31 04:25:55.999835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999755    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:55.999835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999810    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a35242b-41d6-4821-912e-43b4ed6bb53f-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:55.999835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999834    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.000068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:55.999919    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:56.000068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.000042    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:56.002085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.002064    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:25:56.003785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.003772    6931 scope.go:122] "RemoveContainer" containerID="250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"
Oct 31 04:25:56.004012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.003990    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:25:56.004115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.004008    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:56.004649 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:56.004629    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b\": container with ID starting with 250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b not found: ID does not exist" containerID="250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"
Oct 31 04:25:56.004708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.004658    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b"} err="failed to get container status \"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b\": rpc error: code = NotFound desc = could not find container \"250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b\": container with ID starting with 250dd91eb8ce4d5d507666f6aadd550e9d9aa7edd58ded439868b852b2fb682b not found: ID does not exist"
Oct 31 04:25:56.004708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.004676    6931 scope.go:122] "RemoveContainer" containerID="a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"
Oct 31 04:25:56.005241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.005219    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5" pod "1a35242b-41d6-4821-912e-43b4ed6bb53f" (UID: "1a35242b-41d6-4821-912e-43b4ed6bb53f"). InnerVolumeSpecName "kube-api-access-7hqz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:56.014955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.014934    6931 scope.go:122] "RemoveContainer" containerID="a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"
Oct 31 04:25:56.015276 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:25:56.015251    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69\": container with ID starting with a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69 not found: ID does not exist" containerID="a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"
Oct 31 04:25:56.015377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.015283    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69"} err="failed to get container status \"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69\": rpc error: code = NotFound desc = could not find container \"a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69\": container with ID starting with a3fafeae18f20b88ab749b97c3e41d51639a0b303f36fdf2d39eef1f99504e69 not found: ID does not exist"
Oct 31 04:25:56.071135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.062632    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:25:56.071135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.068639    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-801218-5888f5f4c5-5n8nh"]
Oct 31 04:25:56.100986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.100949    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.100986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.100978    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.100986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.100987    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqz5\" (UniqueName: \"kubernetes.io/projected/1a35242b-41d6-4821-912e-43b4ed6bb53f-kube-api-access-7hqz5\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.100986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.100997    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.101581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.101006    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.101581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.101014    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a35242b-41d6-4821-912e-43b4ed6bb53f-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.101581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.101022    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a35242b-41d6-4821-912e-43b4ed6bb53f-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:25:56.129934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.129898    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" path="/var/lib/kubelet/pods/79821c1b-2b14-46c5-aa07-c8e62ce5a355/volumes"
Oct 31 04:25:56.348019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.347751    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:25:56.359810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.359116    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-fsf4c"]
Oct 31 04:25:56.638021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:56.637922    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:57.168481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:57.168445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:57.396315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:57.396280    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:58.123550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:58.123510    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" path="/var/lib/kubelet/pods/1a35242b-41d6-4821-912e-43b4ed6bb53f/volumes"
Oct 31 04:25:58.178662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:58.178623    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:58.396789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:58.396707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:58.919046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:58.919008    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:59.315178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:59.312049    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:59.525645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:25:59.525611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:00.101761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:00.101719    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:01.194051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:01.194017    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:01.980412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:01.980375    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:03.492429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:03.492380    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:05.375459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:05.375415    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:06.377742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:06.377702    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:06.592514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:06.592473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:06.849744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:06.847334    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:07.067626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:07.067586    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:07.275799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:07.275701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:08.151205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.151170    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:26:08.151626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.151384    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" podUID="0a111e24-61f3-4328-9fd8-2d9607435519" containerName="maas-api" containerID="cri-o://db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167" gracePeriod=30
Oct 31 04:26:08.325526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.325494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:08.499123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.499086    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:26:08.590442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.590408    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\") pod \"0a111e24-61f3-4328-9fd8-2d9607435519\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") "
Oct 31 04:26:08.590760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.590737    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\") pod \"0a111e24-61f3-4328-9fd8-2d9607435519\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") "
Oct 31 04:26:08.591013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.590761    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca" pod "0a111e24-61f3-4328-9fd8-2d9607435519" (UID: "0a111e24-61f3-4328-9fd8-2d9607435519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:08.591109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.591088    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\" (UniqueName: \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\") pod \"0a111e24-61f3-4328-9fd8-2d9607435519\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") "
Oct 31 04:26:08.591177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.591135    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\") pod \"0a111e24-61f3-4328-9fd8-2d9607435519\" (UID: \"0a111e24-61f3-4328-9fd8-2d9607435519\") "
Oct 31 04:26:08.591829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.591793    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a111e24-61f3-4328-9fd8-2d9607435519-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:08.600681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.600650    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls" pod "0a111e24-61f3-4328-9fd8-2d9607435519" (UID: "0a111e24-61f3-4328-9fd8-2d9607435519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:08.600845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.600812    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9" pod "0a111e24-61f3-4328-9fd8-2d9607435519" (UID: "0a111e24-61f3-4328-9fd8-2d9607435519"). InnerVolumeSpecName "kube-api-access-696x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:08.601290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.601265    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs" pod "0a111e24-61f3-4328-9fd8-2d9607435519" (UID: "0a111e24-61f3-4328-9fd8-2d9607435519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:08.625491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.625455    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:08.720318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.720107    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:08.720318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.720173    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696x9\" (UniqueName: \"kubernetes.io/projected/0a111e24-61f3-4328-9fd8-2d9607435519-kube-api-access-696x9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:08.720318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:08.720190    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a111e24-61f3-4328-9fd8-2d9607435519-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:09.094274 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.094232    6931 generic.go:356] "Generic (PLEG): container finished" podID="0a111e24-61f3-4328-9fd8-2d9607435519" containerID="db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167" exitCode=0
Oct 31 04:26:09.094461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.094296    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"
Oct 31 04:26:09.094461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.094313    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" event={"ID":"0a111e24-61f3-4328-9fd8-2d9607435519","Type":"ContainerDied","Data":"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"}
Oct 31 04:26:09.094461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.094372    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7" event={"ID":"0a111e24-61f3-4328-9fd8-2d9607435519","Type":"ContainerDied","Data":"218d766cbb5174eba09a1b38eb56392041dbad165c9d680c81edfa86ea08efb1"}
Oct 31 04:26:09.094461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.094399    6931 scope.go:122] "RemoveContainer" containerID="db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"
Oct 31 04:26:09.106002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.105973    6931 scope.go:122] "RemoveContainer" containerID="db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"
Oct 31 04:26:09.106458 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:09.106434    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167\": container with ID starting with db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167 not found: ID does not exist" containerID="db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"
Oct 31 04:26:09.106559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.106471    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167"} err="failed to get container status \"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167\": rpc error: code = NotFound desc = could not find container \"db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167\": container with ID starting with db0f00502ed2cbbd0a0a2a30af09d3fc90b3ec534e899da6e0b1f33cafe30167 not found: ID does not exist"
Oct 31 04:26:09.146408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.146371    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:26:09.158388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.158346    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9154a6-7dbf955574-tlhh7"]
Oct 31 04:26:09.256453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:09.256412    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:10.124123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:10.124090    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a111e24-61f3-4328-9fd8-2d9607435519" path="/var/lib/kubelet/pods/0a111e24-61f3-4328-9fd8-2d9607435519/volumes"
Oct 31 04:26:10.759618 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:10.759583    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:26:11.261424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:11.261381    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:12.655180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:12.655123    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:12.858275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:12.858234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:13.429417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:13.429387    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:13.838806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:13.838772    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:14.052645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:14.052606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:14.779203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:14.779144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:15.045710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:15.045668    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:15.284260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:15.284225    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:15.491520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:15.491424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:16.601440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:16.601364    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:18.539855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:18.539817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:19.540658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:19.540619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:19.755583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:19.755543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:20.660169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:20.660099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:21.365278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:21.365239    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:21.593190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:21.593039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:23.518798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:23.518739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.095753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.095712    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.131588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.131550    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:26:25.131871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.131842    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" podUID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" containerName="payload-pre-processing" containerID="cri-o://fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584" gracePeriod=30
Oct 31 04:26:25.304085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.304044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.515024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.514994    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:26:25.525775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.525634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.665015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.664927    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\") pod \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") "
Oct 31 04:26:25.665183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.665127    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\" (UniqueName: \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\") pod \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\" (UID: \"63510c2c-b057-46d3-ad5c-20c11a1ba97c\") "
Oct 31 04:26:25.665347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.665321    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume" pod "63510c2c-b057-46d3-ad5c-20c11a1ba97c" (UID: "63510c2c-b057-46d3-ad5c-20c11a1ba97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:25.665706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.665686    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63510c2c-b057-46d3-ad5c-20c11a1ba97c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:25.667353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.667334    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg" pod "63510c2c-b057-46d3-ad5c-20c11a1ba97c" (UID: "63510c2c-b057-46d3-ad5c-20c11a1ba97c"). InnerVolumeSpecName "kube-api-access-bn9rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:25.767078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:25.767046    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9rg\" (UniqueName: \"kubernetes.io/projected/63510c2c-b057-46d3-ad5c-20c11a1ba97c-kube-api-access-bn9rg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:26.072302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.072243    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:26.209810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.209774    6931 generic.go:356] "Generic (PLEG): container finished" podID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" containerID="fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584" exitCode=0
Oct 31 04:26:26.210251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.209825    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" event={"ID":"63510c2c-b057-46d3-ad5c-20c11a1ba97c","Type":"ContainerDied","Data":"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"}
Oct 31 04:26:26.210251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.209838    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"
Oct 31 04:26:26.210251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.209866    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks" event={"ID":"63510c2c-b057-46d3-ad5c-20c11a1ba97c","Type":"ContainerDied","Data":"e78e12a8b806c6b8d9cad906099b1074935b1156b4d46fc1d713b40a8425e8f7"}
Oct 31 04:26:26.210251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.209888    6931 scope.go:122] "RemoveContainer" containerID="fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"
Oct 31 04:26:26.219629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.219605    6931 scope.go:122] "RemoveContainer" containerID="fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"
Oct 31 04:26:26.219983 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:26.219952    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584\": container with ID starting with fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584 not found: ID does not exist" containerID="fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"
Oct 31 04:26:26.220054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.219996    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584"} err="failed to get container status \"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584\": rpc error: code = NotFound desc = could not find container \"fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584\": container with ID starting with fd3b70f8cff4dcfcc6ee09c6601cd4927669937b988a4a5be7f7f268435c8584 not found: ID does not exist"
Oct 31 04:26:26.260625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.260591    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:26:26.264955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.264932    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3430f6-759b46c687-rh2ks"]
Oct 31 04:26:26.568858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:26.568822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:27.119601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:27.119571    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c_0ef411f3-5700-4185-821b-6b265c9ba643/payload-processing/0.log"
Oct 31 04:26:27.953927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:27.953885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:28.123049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:28.123017    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" path="/var/lib/kubelet/pods/63510c2c-b057-46d3-ad5c-20c11a1ba97c/volumes"
Oct 31 04:26:29.171453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:29.171417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:30.487716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:30.487662    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:31.292054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:31.292019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:31.517492 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:31.517451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:32.063479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:32.063442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:32.433290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:32.430952    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:32.644584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:32.644546    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:32.899770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:32.899735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:33.233388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:33.233292    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:34.034564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:34.034526    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:26:34.227039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:34.226996    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:35.156003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:35.155967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:36.631623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:36.631591    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:26:36.632174 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:36.631845    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" podUID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" containerName="maas-api" containerID="cri-o://6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9" gracePeriod=30
Oct 31 04:26:36.798079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:36.797990    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:36.952519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:36.952488    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:26:37.007745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.007708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:37.130402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.130373    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\") pod \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") "
Oct 31 04:26:37.130644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.130445    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\") pod \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") "
Oct 31 04:26:37.130644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.130608    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\") pod \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") "
Oct 31 04:26:37.130897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.130711    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\" (UniqueName: \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\") pod \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\" (UID: \"8e0e0f23-da23-458a-8afe-4a82c72a8a6b\") "
Oct 31 04:26:37.131140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.130810    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca" pod "8e0e0f23-da23-458a-8afe-4a82c72a8a6b" (UID: "8e0e0f23-da23-458a-8afe-4a82c72a8a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:37.131394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.131370    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:37.132768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.132741    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs" pod "8e0e0f23-da23-458a-8afe-4a82c72a8a6b" (UID: "8e0e0f23-da23-458a-8afe-4a82c72a8a6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:37.133195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.133169    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6" pod "8e0e0f23-da23-458a-8afe-4a82c72a8a6b" (UID: "8e0e0f23-da23-458a-8afe-4a82c72a8a6b"). InnerVolumeSpecName "kube-api-access-4g2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:37.133195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.133184    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls" pod "8e0e0f23-da23-458a-8afe-4a82c72a8a6b" (UID: "8e0e0f23-da23-458a-8afe-4a82c72a8a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:37.237589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.237050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:37.238741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.238715    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2c6\" (UniqueName: \"kubernetes.io/projected/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-kube-api-access-4g2c6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:37.238910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.238895    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:37.239011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.239000    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e0e0f23-da23-458a-8afe-4a82c72a8a6b-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:37.290208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.287543    6931 generic.go:356] "Generic (PLEG): container finished" podID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" containerID="6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9" exitCode=0
Oct 31 04:26:37.290208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.287719    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" event={"ID":"8e0e0f23-da23-458a-8afe-4a82c72a8a6b","Type":"ContainerDied","Data":"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"}
Oct 31 04:26:37.290208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.287764    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw" event={"ID":"8e0e0f23-da23-458a-8afe-4a82c72a8a6b","Type":"ContainerDied","Data":"85b7aacabb9a222ee7829348e30acb5544f8a2b63a5a0f2571133fc4df4c6941"}
Oct 31 04:26:37.290208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.287784    6931 scope.go:122] "RemoveContainer" containerID="6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"
Oct 31 04:26:37.290208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.287901    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"
Oct 31 04:26:37.303026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.303004    6931 scope.go:122] "RemoveContainer" containerID="6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"
Oct 31 04:26:37.304223 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:37.304197    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9\": container with ID starting with 6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9 not found: ID does not exist" containerID="6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"
Oct 31 04:26:37.304311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.304230    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9"} err="failed to get container status \"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9\": rpc error: code = NotFound desc = could not find container \"6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9\": container with ID starting with 6b3cbad4b160a43677a90d404e3fa379d429b70642ee9856f77c8e272c14c8e9 not found: ID does not exist"
Oct 31 04:26:37.348096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.348058    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:26:37.357357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.357325    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ceba51-577c4f87dc-kq5pw"]
Oct 31 04:26:37.923730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:37.923693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:38.122259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:38.122216    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" path="/var/lib/kubelet/pods/8e0e0f23-da23-458a-8afe-4a82c72a8a6b/volumes"
Oct 31 04:26:39.010192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:39.010142    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:39.309504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:39.309462    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:39.846905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:39.846869    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:40.053737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:40.053706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:40.289723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:40.281043    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:40.648217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:40.648085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:40.758895 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:40.758856    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:26:41.045110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:41.045076    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:42.422964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:42.422931    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:42.637214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:42.636309    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:44.364069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:44.364034    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:45.738551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:45.738519    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:26:45.739027 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:45.738998    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" podUID="e6da865f-eb7f-4419-a6f0-8642fa00221b" containerName="maas-api" containerID="cri-o://90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb" gracePeriod=30
Oct 31 04:26:45.909076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:45.909039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:46.096058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.096032    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:26:46.123901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.123868    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:46.177298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.177260    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") pod \"e6da865f-eb7f-4419-a6f0-8642fa00221b\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") "
Oct 31 04:26:46.177496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.177371    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\") pod \"e6da865f-eb7f-4419-a6f0-8642fa00221b\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") "
Oct 31 04:26:46.177561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.177505    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\" (UniqueName: \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\") pod \"e6da865f-eb7f-4419-a6f0-8642fa00221b\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") "
Oct 31 04:26:46.177561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.177545    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\") pod \"e6da865f-eb7f-4419-a6f0-8642fa00221b\" (UID: \"e6da865f-eb7f-4419-a6f0-8642fa00221b\") "
Oct 31 04:26:46.178005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.177974    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca" pod "e6da865f-eb7f-4419-a6f0-8642fa00221b" (UID: "e6da865f-eb7f-4419-a6f0-8642fa00221b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:46.178146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.178121    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6da865f-eb7f-4419-a6f0-8642fa00221b-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:46.179684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.179660    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs" pod "e6da865f-eb7f-4419-a6f0-8642fa00221b" (UID: "e6da865f-eb7f-4419-a6f0-8642fa00221b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:46.179910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.179890    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls" pod "e6da865f-eb7f-4419-a6f0-8642fa00221b" (UID: "e6da865f-eb7f-4419-a6f0-8642fa00221b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:46.179988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.179902    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j" pod "e6da865f-eb7f-4419-a6f0-8642fa00221b" (UID: "e6da865f-eb7f-4419-a6f0-8642fa00221b"). InnerVolumeSpecName "kube-api-access-7nj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:46.279630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.279589    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:46.279630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.279620    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nj4j\" (UniqueName: \"kubernetes.io/projected/e6da865f-eb7f-4419-a6f0-8642fa00221b-kube-api-access-7nj4j\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:46.279630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.279629    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6da865f-eb7f-4419-a6f0-8642fa00221b-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:46.332239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.331930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:46.380715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.380673    6931 generic.go:356] "Generic (PLEG): container finished" podID="e6da865f-eb7f-4419-a6f0-8642fa00221b" containerID="90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb" exitCode=0
Oct 31 04:26:46.380922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.380776    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" event={"ID":"e6da865f-eb7f-4419-a6f0-8642fa00221b","Type":"ContainerDied","Data":"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"}
Oct 31 04:26:46.380922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.380827    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w" event={"ID":"e6da865f-eb7f-4419-a6f0-8642fa00221b","Type":"ContainerDied","Data":"9ff583ee6e8438f5b4e99af740657885b1f3b2287db7debc4d498e5db7450ac1"}
Oct 31 04:26:46.380922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.380851    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"
Oct 31 04:26:46.380922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.380857    6931 scope.go:122] "RemoveContainer" containerID="90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"
Oct 31 04:26:46.396040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.396018    6931 scope.go:122] "RemoveContainer" containerID="90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"
Oct 31 04:26:46.396426 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:46.396404    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb\": container with ID starting with 90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb not found: ID does not exist" containerID="90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"
Oct 31 04:26:46.396507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.396441    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb"} err="failed to get container status \"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb\": rpc error: code = NotFound desc = could not find container \"90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb\": container with ID starting with 90043cc8b524780577d56f728f038b81b533e38865646e231246c1cfbc5739fb not found: ID does not exist"
Oct 31 04:26:46.442626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.442587    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:26:46.448756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.448713    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b4185-b684859db-8m84w"]
Oct 31 04:26:46.746950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:46.746866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:47.471409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:47.471367    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:48.122243 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:48.122209    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6da865f-eb7f-4419-a6f0-8642fa00221b" path="/var/lib/kubelet/pods/e6da865f-eb7f-4419-a6f0-8642fa00221b/volumes"
Oct 31 04:26:48.868223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:48.868185    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:49.056173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.056120    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" podUID="8db1f49a-3c83-4899-b9cf-dc07701def71" containerName="main" containerID="cri-o://15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a" gracePeriod=45
Oct 31 04:26:49.330956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.330928    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:26:49.402347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.402312    6931 generic.go:356] "Generic (PLEG): container finished" podID="8db1f49a-3c83-4899-b9cf-dc07701def71" containerID="15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a" exitCode=0
Oct 31 04:26:49.402538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.402384    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"
Oct 31 04:26:49.402538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.402396    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" event={"ID":"8db1f49a-3c83-4899-b9cf-dc07701def71","Type":"ContainerDied","Data":"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"}
Oct 31 04:26:49.402538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.402458    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc" event={"ID":"8db1f49a-3c83-4899-b9cf-dc07701def71","Type":"ContainerDied","Data":"462ec926bdf78fb48e35a0a205d87d2bff10c7f09148bfbafb64f3f8e485fcb0"}
Oct 31 04:26:49.402538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.402479    6931 scope.go:122] "RemoveContainer" containerID="15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"
Oct 31 04:26:49.416088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.416059    6931 scope.go:122] "RemoveContainer" containerID="15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"
Oct 31 04:26:49.418076 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:49.417965    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a\": container with ID starting with 15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a not found: ID does not exist" containerID="15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"
Oct 31 04:26:49.418213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.418090    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a"} err="failed to get container status \"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a\": rpc error: code = NotFound desc = could not find container \"15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a\": container with ID starting with 15d4a055af99b4c5317d620e2d69f4ab62ff870f4b24489a8d88c79b7cbd607a not found: ID does not exist"
Oct 31 04:26:49.420751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.420724    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.420907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.420889    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\" (UniqueName: \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.421053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421037    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.421385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421345    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:49.421490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421402    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.421561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421511    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.421619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421583    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\" (UniqueName: \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\") pod \"8db1f49a-3c83-4899-b9cf-dc07701def71\" (UID: \"8db1f49a-3c83-4899-b9cf-dc07701def71\") "
Oct 31 04:26:49.422034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421890    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:49.422034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.421942    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:49.422204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.422108    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.422204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.422128    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.422204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.422203    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.426238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.424929    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:49.426238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.425297    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "kube-api-access-x47d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:49.427286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.427262    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm" pod "8db1f49a-3c83-4899-b9cf-dc07701def71" (UID: "8db1f49a-3c83-4899-b9cf-dc07701def71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:49.524021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.523976    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47d4\" (UniqueName: \"kubernetes.io/projected/8db1f49a-3c83-4899-b9cf-dc07701def71-kube-api-access-x47d4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.524021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.524014    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db1f49a-3c83-4899-b9cf-dc07701def71-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.524021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.524030    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db1f49a-3c83-4899-b9cf-dc07701def71-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:49.748534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.748501    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:26:49.752145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:49.752122    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f91cd1/ipp-route-f91cd1-kserve-858658d885-g9jjc"]
Oct 31 04:26:50.124214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:50.124171    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db1f49a-3c83-4899-b9cf-dc07701def71" path="/var/lib/kubelet/pods/8db1f49a-3c83-4899-b9cf-dc07701def71/volumes"
Oct 31 04:26:50.739610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:50.739503    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:53.098804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:53.098770    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:54.477511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.477477    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:26:54.478229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.478197    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" podUID="416e232e-c795-4209-982a-08405291ede8" containerName="maas-api" containerID="cri-o://178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54" gracePeriod=30
Oct 31 04:26:54.640020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.639922    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:54.809018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.808986    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:26:54.857173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.857129    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:54.901363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.901320    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\" (UniqueName: \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\") pod \"416e232e-c795-4209-982a-08405291ede8\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") "
Oct 31 04:26:54.901530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.901404    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\") pod \"416e232e-c795-4209-982a-08405291ede8\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") "
Oct 31 04:26:54.901530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.901520    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") pod \"416e232e-c795-4209-982a-08405291ede8\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") "
Oct 31 04:26:54.901625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.901607    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\") pod \"416e232e-c795-4209-982a-08405291ede8\" (UID: \"416e232e-c795-4209-982a-08405291ede8\") "
Oct 31 04:26:54.901961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.901934    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca" pod "416e232e-c795-4209-982a-08405291ede8" (UID: "416e232e-c795-4209-982a-08405291ede8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:54.903572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.903542    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls" pod "416e232e-c795-4209-982a-08405291ede8" (UID: "416e232e-c795-4209-982a-08405291ede8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:54.903572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.903556    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs" pod "416e232e-c795-4209-982a-08405291ede8" (UID: "416e232e-c795-4209-982a-08405291ede8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:54.903706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:54.903672    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w" pod "416e232e-c795-4209-982a-08405291ede8" (UID: "416e232e-c795-4209-982a-08405291ede8"). InnerVolumeSpecName "kube-api-access-cxn6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:55.006499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.006457    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:55.006499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.006499    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416e232e-c795-4209-982a-08405291ede8-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:55.006778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.006513    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416e232e-c795-4209-982a-08405291ede8-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:55.006778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.006528    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxn6w\" (UniqueName: \"kubernetes.io/projected/416e232e-c795-4209-982a-08405291ede8-kube-api-access-cxn6w\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:26:55.071382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.071341    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:55.276504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.276469    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:55.445416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.445343    6931 generic.go:356] "Generic (PLEG): container finished" podID="416e232e-c795-4209-982a-08405291ede8" containerID="178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54" exitCode=0
Oct 31 04:26:55.445416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.445399    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"
Oct 31 04:26:55.445580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.445434    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" event={"ID":"416e232e-c795-4209-982a-08405291ede8","Type":"ContainerDied","Data":"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"}
Oct 31 04:26:55.445580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.445502    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r" event={"ID":"416e232e-c795-4209-982a-08405291ede8","Type":"ContainerDied","Data":"ac0623b7576e7aefd98b23f4cf1017570fa1e00488f3b5c614b4a2b939e36d4d"}
Oct 31 04:26:55.445580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.445537    6931 scope.go:122] "RemoveContainer" containerID="178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"
Oct 31 04:26:55.456149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.456125    6931 scope.go:122] "RemoveContainer" containerID="178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"
Oct 31 04:26:55.456475 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:26:55.456455    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54\": container with ID starting with 178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54 not found: ID does not exist" containerID="178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"
Oct 31 04:26:55.456535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.456488    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54"} err="failed to get container status \"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54\": rpc error: code = NotFound desc = could not find container \"178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54\": container with ID starting with 178225c68e84737e32a22cc748568e418df5168b94fe4a2a4a80a9f90ee97b54 not found: ID does not exist"
Oct 31 04:26:55.496503 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.496472    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:26:55.509654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.509624    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9b573-8547fd8694-2rk8r"]
Oct 31 04:26:55.667406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:55.667366    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:56.083585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:56.083546    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:56.123324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:56.123289    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416e232e-c795-4209-982a-08405291ede8" path="/var/lib/kubelet/pods/416e232e-c795-4209-982a-08405291ede8/volumes"
Oct 31 04:26:56.404133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:56.390610    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:57.344292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:57.344247    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:57.784308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:57.784276    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:58.004391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:58.004350    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:58.347176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:58.343350    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:58.568214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:58.566547    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:58.776565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:58.776468    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:59.172177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:59.171873    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:59.936570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:26:59.936533    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:00.399842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:00.399787    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:02.124837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:02.124794    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:02.339582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:02.339549    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:02.562778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:02.562738    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:02.962984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:02.962903    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:03.199628 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:03.199593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:03.406323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:03.406288    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:03.847185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:03.847116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:04.063032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:04.062915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:04.269998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:04.269909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:04.529829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:04.529789    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:04.812525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:04.812435    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:05.546996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:05.546962    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:05.934274 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:05.934165    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:06.148874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:06.148826    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:06.918226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:06.915760    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:07.140237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:07.140189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:07.384530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:07.384490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:07.853176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:07.850812    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:08.149197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:08.149044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:09.101894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:09.101863    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:10.487600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:10.487557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:10.759068 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:27:10.758979    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:27:11.151441 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:11.151402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:12.017897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:12.017856    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:12.233936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:12.233896    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:12.622021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:12.621980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:13.313991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:13.313950    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:13.735172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:13.735077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:14.014480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:14.014408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:15.545050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:15.545005    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:17.995543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:17.995498    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:18.503808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:18.503764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:19.767505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:19.767470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:19.982563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:19.982526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:21.722667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:21.722630    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:21.936048 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:21.935990    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:23.695494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:23.695452    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:24.775084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:24.775047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:25.049261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:25.049219    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:25.261072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:25.261015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:26.396686 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:26.396647    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:28.231007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:28.230968    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:29.000152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:29.000112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:30.044883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:30.044847    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:32.041584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:32.041518    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:32.283947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:32.283905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:33.530785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.530749    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:27:33.531429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.530950    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" podUID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" containerName="maas-api" containerID="cri-o://2540c5f7d767bb608e2521736aa3337795cb85e3e02df520eed83df76a569914" gracePeriod=30
Oct 31 04:27:33.703830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.703791    6931 generic.go:356] "Generic (PLEG): container finished" podID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" containerID="2540c5f7d767bb608e2521736aa3337795cb85e3e02df520eed83df76a569914" exitCode=0
Oct 31 04:27:33.704027 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.703883    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" event={"ID":"10bab208-0bae-4a6c-a50d-7fbe8c5825e8","Type":"ContainerDied","Data":"2540c5f7d767bb608e2521736aa3337795cb85e3e02df520eed83df76a569914"}
Oct 31 04:27:33.705555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.705524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:33.706510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.706483    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:27:33.706863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.706836    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" podUID="0ef411f3-5700-4185-821b-6b265c9ba643" containerName="payload-processing" containerID="cri-o://053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb" gracePeriod=30
Oct 31 04:27:33.884714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.884556    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:27:33.937428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.937208    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:33.984802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.984765    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\") pod \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") "
Oct 31 04:27:33.984954 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.984864    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\" (UniqueName: \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\") pod \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") "
Oct 31 04:27:33.985061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.985038    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\") pod \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") "
Oct 31 04:27:33.985221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.985081    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\") pod \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\" (UID: \"10bab208-0bae-4a6c-a50d-7fbe8c5825e8\") "
Oct 31 04:27:33.985453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.985429    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca" pod "10bab208-0bae-4a6c-a50d-7fbe8c5825e8" (UID: "10bab208-0bae-4a6c-a50d-7fbe8c5825e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:33.985639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.985609    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:33.987390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.987363    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls" pod "10bab208-0bae-4a6c-a50d-7fbe8c5825e8" (UID: "10bab208-0bae-4a6c-a50d-7fbe8c5825e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:33.987495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.987469    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs" pod "10bab208-0bae-4a6c-a50d-7fbe8c5825e8" (UID: "10bab208-0bae-4a6c-a50d-7fbe8c5825e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:33.987662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:33.987637    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q" pod "10bab208-0bae-4a6c-a50d-7fbe8c5825e8" (UID: "10bab208-0bae-4a6c-a50d-7fbe8c5825e8"). InnerVolumeSpecName "kube-api-access-68d2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:34.003121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.003090    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:27:34.114115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.112654    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\" (UniqueName: \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\") pod \"0ef411f3-5700-4185-821b-6b265c9ba643\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") "
Oct 31 04:27:34.114546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.114522    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\") pod \"0ef411f3-5700-4185-821b-6b265c9ba643\" (UID: \"0ef411f3-5700-4185-821b-6b265c9ba643\") "
Oct 31 04:27:34.122214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.122184    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:34.122529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.122509    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68d2q\" (UniqueName: \"kubernetes.io/projected/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-kube-api-access-68d2q\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:34.122658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.122643    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10bab208-0bae-4a6c-a50d-7fbe8c5825e8-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:34.123006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.122987    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume" pod "0ef411f3-5700-4185-821b-6b265c9ba643" (UID: "0ef411f3-5700-4185-821b-6b265c9ba643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:34.138534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.138495    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd" pod "0ef411f3-5700-4185-821b-6b265c9ba643" (UID: "0ef411f3-5700-4185-821b-6b265c9ba643"). InnerVolumeSpecName "kube-api-access-4pxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:34.223989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.223953    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef411f3-5700-4185-821b-6b265c9ba643-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:34.223989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.223989    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxbd\" (UniqueName: \"kubernetes.io/projected/0ef411f3-5700-4185-821b-6b265c9ba643-kube-api-access-4pxbd\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:27:34.723520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.723475    6931 generic.go:356] "Generic (PLEG): container finished" podID="0ef411f3-5700-4185-821b-6b265c9ba643" containerID="053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb" exitCode=0
Oct 31 04:27:34.724011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.723543    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"
Oct 31 04:27:34.724011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.723547    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" event={"ID":"0ef411f3-5700-4185-821b-6b265c9ba643","Type":"ContainerDied","Data":"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"}
Oct 31 04:27:34.724011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.723632    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c" event={"ID":"0ef411f3-5700-4185-821b-6b265c9ba643","Type":"ContainerDied","Data":"a0f9aac2317c376df50e1adea95b987988d3af487dabe87f69c4f6d3607cf3b6"}
Oct 31 04:27:34.724011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.723665    6931 scope.go:122] "RemoveContainer" containerID="053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"
Oct 31 04:27:34.726873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.726844    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6" event={"ID":"10bab208-0bae-4a6c-a50d-7fbe8c5825e8","Type":"ContainerDied","Data":"7e17f15a816550655a76b0215d3e39c8321852a7f7ad124c87b9a9f4559c04e2"}
Oct 31 04:27:34.726985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.726920    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"
Oct 31 04:27:34.736231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.736213    6931 scope.go:122] "RemoveContainer" containerID="053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"
Oct 31 04:27:34.736499 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:27:34.736475    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb\": container with ID starting with 053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb not found: ID does not exist" containerID="053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"
Oct 31 04:27:34.736598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.736506    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb"} err="failed to get container status \"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb\": rpc error: code = NotFound desc = could not find container \"053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb\": container with ID starting with 053f6a7fbcd837eb53cbec6200565d433e3ed21fdb855dafc812beb80b2da3eb not found: ID does not exist"
Oct 31 04:27:34.736598 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.736526    6931 scope.go:122] "RemoveContainer" containerID="2540c5f7d767bb608e2521736aa3337795cb85e3e02df520eed83df76a569914"
Oct 31 04:27:34.775215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.775180    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:27:34.780471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.780443    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f91cd1-64f6c8d49d-phwq6"]
Oct 31 04:27:34.814065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.813984    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:27:34.820808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:34.820761    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f91cd1-7d754d98bb-h696c"]
Oct 31 04:27:36.122472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:36.122441    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef411f3-5700-4185-821b-6b265c9ba643" path="/var/lib/kubelet/pods/0ef411f3-5700-4185-821b-6b265c9ba643/volumes"
Oct 31 04:27:36.122884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:36.122870    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" path="/var/lib/kubelet/pods/10bab208-0bae-4a6c-a50d-7fbe8c5825e8/volumes"
Oct 31 04:27:36.268250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:36.268209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:36.484108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:36.484023    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:38.139060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:38.139021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:40.442360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:40.442323    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:40.759803 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:27:40.759714    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:27:41.153324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:41.153209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:41.451414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:41.451318    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:41.890302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:41.890265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:42.300789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:42.300749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:42.521771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:42.521739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:42.756600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:42.756561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:43.029755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:43.029663    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:43.258378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:43.258340    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:44.514729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:44.514641    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:46.721837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:46.721803    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:46.951211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:46.951145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:47.299627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:47.299590    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:47.786190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:47.786116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:47.996699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:47.996664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:48.692727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:48.692693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:48.929022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:48.928976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:49.148873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:49.148834    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:49.411625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:49.411524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:49.659571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:49.659534    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:50.003293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:50.003259    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:50.500435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:50.495013    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:50.706821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:50.706752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:51.512325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:51.512274    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:53.187856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:53.187819    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:53.416111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:53.416072    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:53.625512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:53.625476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:54.792623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:54.792571    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:55.041188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:55.041021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:55.550827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:55.550791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:55.781462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:55.781426    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:56.639471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:56.639434    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:56.858990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:56.858909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:57.423455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:57.423416    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:57.829197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:57.829132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:59.908442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:27:59.908402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:00.117974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:00.117936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.445469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:01.445428    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.654905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:01.654873    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.908268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:01.908207    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:02.390728 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:02.390696    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:02.917538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:02.917500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:04.110743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:04.110706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:04.325933 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:04.325853    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:04.590507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:04.590471    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:04.800016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:04.799958    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:06.394664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:06.394619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:08.187300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187264    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:28:08.187848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187834    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy"
Oct 31 04:28:08.187892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187850    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy"
Oct 31 04:28:08.187892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187859    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0ef411f3-5700-4185-821b-6b265c9ba643" containerName="payload-processing"
Oct 31 04:28:08.187892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187865    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef411f3-5700-4185-821b-6b265c9ba643" containerName="payload-processing"
Oct 31 04:28:08.187892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187881    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0a111e24-61f3-4328-9fd8-2d9607435519" containerName="maas-api"
Oct 31 04:28:08.187892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187889    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a111e24-61f3-4328-9fd8-2d9607435519" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187902    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8db1f49a-3c83-4899-b9cf-dc07701def71" containerName="main"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187908    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db1f49a-3c83-4899-b9cf-dc07701def71" containerName="main"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187915    6931 cpu_manager.go:414] "containerMap: removing container" podUID="e6da865f-eb7f-4419-a6f0-8642fa00221b" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187920    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6da865f-eb7f-4419-a6f0-8642fa00221b" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187928    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187932    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187943    6931 cpu_manager.go:414] "containerMap: removing container" podUID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" containerName="payload-pre-processing"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187947    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" containerName="payload-pre-processing"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187959    6931 cpu_manager.go:414] "containerMap: removing container" podUID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187967    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187979    6931 cpu_manager.go:414] "containerMap: removing container" podUID="416e232e-c795-4209-982a-08405291ede8" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187983    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416e232e-c795-4209-982a-08405291ede8" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.187999    6931 cpu_manager.go:414] "containerMap: removing container" podUID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188003    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" containerName="maas-api"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188013    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy"
Oct 31 04:28:08.188045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188018    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188103    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f51be56-c3fd-4fe1-a020-cd45da9043d5" containerName="istio-proxy"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188118    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a35242b-41d6-4821-912e-43b4ed6bb53f" containerName="istio-proxy"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188128    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef411f3-5700-4185-821b-6b265c9ba643" containerName="payload-processing"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188135    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a111e24-61f3-4328-9fd8-2d9607435519" containerName="maas-api"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188142    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db1f49a-3c83-4899-b9cf-dc07701def71" containerName="main"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188147    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6da865f-eb7f-4419-a6f0-8642fa00221b" containerName="maas-api"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188164    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0e0f23-da23-458a-8afe-4a82c72a8a6b" containerName="maas-api"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188175    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="63510c2c-b057-46d3-ad5c-20c11a1ba97c" containerName="payload-pre-processing"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188183    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="79821c1b-2b14-46c5-aa07-c8e62ce5a355" containerName="maas-api"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188194    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="416e232e-c795-4209-982a-08405291ede8" containerName="maas-api"
Oct 31 04:28:08.188527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.188205    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="10bab208-0bae-4a6c-a50d-7fbe8c5825e8" containerName="maas-api"
Oct 31 04:28:08.192829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.192811    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.225526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.225487    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-24cd29-serving-cert\""
Oct 31 04:28:08.229736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.229708    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:28:08.293741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.293706    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.293923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.293765    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbhj\" (UniqueName: \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.293923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.293907    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.294003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.293939    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.387318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.387268    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:08.395654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.395627    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.395745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.395695    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbhj\" (UniqueName: \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.395809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.395793    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.395850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.395834    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.396406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.396385    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.398137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.398115    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.398257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.398219    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.435977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.435944    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbhj\" (UniqueName: \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\") pod \"maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.505691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.505600    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:08.593641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.593582    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:08.806586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.806550    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:08.812737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.812710    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:28:08.816759 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:28:08.816722    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce5b5cd_695f_41fb_96e2_ab561cc93aec.slice/crio-be5b79248a52c04cc9497cd5515f6596c17e3df7435129f935d95556f3ca46be WatchSource:0}: Error finding container be5b79248a52c04cc9497cd5515f6596c17e3df7435129f935d95556f3ca46be: Status 404 returned error can't find the container with id be5b79248a52c04cc9497cd5515f6596c17e3df7435129f935d95556f3ca46be
Oct 31 04:28:08.818139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:08.818120    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:28:09.015581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:09.015484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:09.018190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:09.018136    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" event={"ID":"1ce5b5cd-695f-41fb-96e2-ab561cc93aec","Type":"ContainerStarted","Data":"be5b79248a52c04cc9497cd5515f6596c17e3df7435129f935d95556f3ca46be"}
Oct 31 04:28:10.032020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:10.031771    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" event={"ID":"1ce5b5cd-695f-41fb-96e2-ab561cc93aec","Type":"ContainerStarted","Data":"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"}
Oct 31 04:28:10.033878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:10.033849    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:10.081827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:10.081763    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" podStartSLOduration=1.6817721319999999 podStartE2EDuration="2.081746235s" totalImagesPullingTime="399.974103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:10.077875557 +0000 UTC m=+1386.110875236" watchObservedRunningTime="2026-10-31 04:28:10.081746235 +0000 UTC m=+1386.114745913"
Oct 31 04:28:10.433449 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:10.433344    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:10.652796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:10.652722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:10.770437 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:28:10.770349    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:28:12.141376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:12.141329    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:12.430579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:12.430494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:13.043741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:13.043699    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:13.263796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:13.263756    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:13.479134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:13.479046    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:14.032916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:14.032883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:14.240565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:14.240526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:14.954533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:14.954494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:15.219056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:15.218971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:15.527560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:15.527463    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:15.740622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:15.740584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:16.980539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:16.980502    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:28:16.984637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:16.984621    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.004611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.004577    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:28:17.013136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.013100    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b44631-serving-cert\""
Oct 31 04:28:17.057341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.057301    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.057504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.057351    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.057504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.057402    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.057504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.057498    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5hz\" (UniqueName: \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.058025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.058002    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:28:17.158981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.158929    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5hz\" (UniqueName: \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.159267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.159202    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.159267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.159241    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.159414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.159304    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.160377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.160340    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.162621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.162597    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.162760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.162740    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.174256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.174224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:17.182439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.182415    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5hz\" (UniqueName: \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\") pod \"maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.298136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.298106    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:17.399185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.395691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:17.620571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:17.620487    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:28:17.623932 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:28:17.623906    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccbec93_da10_49ec_9e23_6fb2f9f1aaf6.slice/crio-88a5dbc5726bbbd65dafa84455bb06a29edd4b9c64fb2899d47184c19de20c9e WatchSource:0}: Error finding container 88a5dbc5726bbbd65dafa84455bb06a29edd4b9c64fb2899d47184c19de20c9e: Status 404 returned error can't find the container with id 88a5dbc5726bbbd65dafa84455bb06a29edd4b9c64fb2899d47184c19de20c9e
Oct 31 04:28:18.081624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:18.081580    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" event={"ID":"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6","Type":"ContainerStarted","Data":"a554f78fd4d81787ada083104e356eaf7e1763afb0359d6adad2e573fcc4d66f"}
Oct 31 04:28:18.082072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:18.081652    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" event={"ID":"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6","Type":"ContainerStarted","Data":"88a5dbc5726bbbd65dafa84455bb06a29edd4b9c64fb2899d47184c19de20c9e"}
Oct 31 04:28:18.082212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:18.082150    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:18.127450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:18.127387    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" podStartSLOduration=1.7973856160000001 podStartE2EDuration="2.127369657s" totalImagesPullingTime="329.984041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:18.124712436 +0000 UTC m=+1394.157712117" watchObservedRunningTime="2026-10-31 04:28:18.127369657 +0000 UTC m=+1394.160369336"
Oct 31 04:28:19.257549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:19.257509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:19.469477 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:19.469442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:19.681345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:19.681311    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:19.911818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:19.911781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:21.376636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:21.376594    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:21.621178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:21.616347    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:21.846720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:21.846634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:23.799727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:23.799679    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:24.097114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:24.097028    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:28:25.530205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:25.530128    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:27.324331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:27.324293    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:29.192007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:29.191968    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:30.986016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:30.985980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:32.721661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:32.721607    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:34.367626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.367579    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:28:34.373585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.373553    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.389793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.389763    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:28:34.398075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.398047    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-81c810-serving-cert\""
Oct 31 04:28:34.505521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.505477    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jxx\" (UniqueName: \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.505737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.505619    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.505815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.505743    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.505873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.505835    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.556489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.556457    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:34.607108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.607068    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.607310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.607185    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.607373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.607347    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.607429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.607410    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jxx\" (UniqueName: \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.607972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.607939    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.609640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.609620    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.609739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.609670    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.652663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.652577    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jxx\" (UniqueName: \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\") pod \"maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.689038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.688995    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:34.771003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.770957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:34.993240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:34.993118    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:28:34.997040 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:28:34.996977    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec7a65f_5ab6_4e44_b005_ad7ffae8943c.slice/crio-7b062511dc478830d5a7a95b6a1992b74b4b5e98a580da2e78876fba5dd1b14f WatchSource:0}: Error finding container 7b062511dc478830d5a7a95b6a1992b74b4b5e98a580da2e78876fba5dd1b14f: Status 404 returned error can't find the container with id 7b062511dc478830d5a7a95b6a1992b74b4b5e98a580da2e78876fba5dd1b14f
Oct 31 04:28:35.204937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:35.204022    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" event={"ID":"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c","Type":"ContainerStarted","Data":"7b062511dc478830d5a7a95b6a1992b74b4b5e98a580da2e78876fba5dd1b14f"}
Oct 31 04:28:36.211941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:36.211899    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" event={"ID":"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c","Type":"ContainerStarted","Data":"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"}
Oct 31 04:28:36.212529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:36.211965    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:36.255137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:36.255078    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" podStartSLOduration=1.953482699 podStartE2EDuration="2.255063748s" totalImagesPullingTime="301.581049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:36.252792492 +0000 UTC m=+1412.285792172" watchObservedRunningTime="2026-10-31 04:28:36.255063748 +0000 UTC m=+1412.288063426"
Oct 31 04:28:36.503281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:36.503196    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:38.531982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:38.531935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:40.760301 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:28:40.760262    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:28:41.098751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:41.098709    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:42.224346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:42.224311    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:28:43.003226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:43.003088    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:43.218641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:43.218599    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:43.427259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:43.427215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:45.067296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:45.067262    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:46.907219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:46.907175    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:47.127419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:47.127369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:47.357613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:47.357573    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:49.057819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:49.057787    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:49.507695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:49.507574    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:51.168712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:51.168669    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:51.688396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:51.688359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:53.648267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:53.648229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:55.333393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:55.333347    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:55.560911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:55.560868    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:56.856012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:56.855936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:57.087208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:57.081082    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:57.542707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:57.542673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:57.754112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:57.754069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:59.741809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:28:59.741757    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:00.017905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:00.017811    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:01.797554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:01.797513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:02.849824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:02.849782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:03.953039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:03.952996    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:04.163382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:04.163343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:04.815747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:04.815706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:05.045560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:05.045526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:05.255664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:05.255580    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:06.034810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:06.034769    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:06.463316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:06.463223    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:07.295659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:07.295606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:08.627222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:08.627143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:08.905018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:08.904918    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:09.823713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:09.823675    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:10.050275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:10.050245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:10.763583 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:29:10.763543    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:29:12.464830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:12.464791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:12.672823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:12.672785    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:13.199194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:13.199137    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:13.511501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:13.511408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:13.724006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:13.723953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:14.063461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:14.063426    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:14.776564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:14.776519    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:16.164573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.164539    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:29:16.165035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.164745    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" podUID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" containerName="maas-api" containerID="cri-o://a554f78fd4d81787ada083104e356eaf7e1763afb0359d6adad2e573fcc4d66f" gracePeriod=30
Oct 31 04:29:16.329198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.329102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:16.525603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.525486    6931 generic.go:356] "Generic (PLEG): container finished" podID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" containerID="a554f78fd4d81787ada083104e356eaf7e1763afb0359d6adad2e573fcc4d66f" exitCode=0
Oct 31 04:29:16.525603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.525521    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" event={"ID":"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6","Type":"ContainerDied","Data":"a554f78fd4d81787ada083104e356eaf7e1763afb0359d6adad2e573fcc4d66f"}
Oct 31 04:29:16.547848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.547816    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:29:16.698509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.698469    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\") pod \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") "
Oct 31 04:29:16.698879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.698863    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\" (UniqueName: \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\") pod \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") "
Oct 31 04:29:16.699097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.699080    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\") pod \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") "
Oct 31 04:29:16.699275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.699257    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\") pod \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\" (UID: \"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6\") "
Oct 31 04:29:16.707715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.707663    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs" pod "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" (UID: "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:29:16.711455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.711399    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca" pod "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" (UID: "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:16.719951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.719915    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz" pod "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" (UID: "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6"). InnerVolumeSpecName "kube-api-access-zw5hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:16.724354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.723301    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls" pod "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" (UID: "7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:29:16.800972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.800943    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5hz\" (UniqueName: \"kubernetes.io/projected/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-kube-api-access-zw5hz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:16.800972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.800968    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:16.800972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.800978    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:16.801239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:16.800986    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:17.152840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.152754    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:17.532876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.532850    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"
Oct 31 04:29:17.532876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.532860    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4" event={"ID":"7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6","Type":"ContainerDied","Data":"88a5dbc5726bbbd65dafa84455bb06a29edd4b9c64fb2899d47184c19de20c9e"}
Oct 31 04:29:17.533455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.532915    6931 scope.go:122] "RemoveContainer" containerID="a554f78fd4d81787ada083104e356eaf7e1763afb0359d6adad2e573fcc4d66f"
Oct 31 04:29:17.583721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.583685    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:29:17.595970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:17.595935    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b44631-556575d95c-5cxw4"]
Oct 31 04:29:18.122446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:18.122412    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" path="/var/lib/kubelet/pods/7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6/volumes"
Oct 31 04:29:19.032808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:19.032748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:19.456298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:19.456196    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:19.672226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:19.672179    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:20.200316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:20.200272    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:20.615206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:20.611392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:20.824818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:20.824778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:21.207518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:21.207479    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:21.989797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:21.989745    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:22.468181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:22.467868    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:22.909724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:22.909691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:23.157804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:23.157767    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:25.106746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:25.106659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:26.555772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:26.555736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:26.782717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:26.782680    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:27.241354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:27.241315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:28.318094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:28.318061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:28.530021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:28.529843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:29.215653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:29.215622    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:29.429023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:29.428988    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:30.783046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:30.783007    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:31.719579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:31.719543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:33.620095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:33.620059    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:35.324242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:35.324207    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:35.802633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:35.802584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:36.033259 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:36.033209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:36.245481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:36.245355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:36.767660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:36.767563    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:38.610007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:38.609958    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:38.822050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:38.822009    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:39.472967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:39.472934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:39.865855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:39.865818    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:40.612929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:40.612892    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:40.758954 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:29:40.758916    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:29:40.822540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:40.822500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:41.263326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:41.263278    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:41.482976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:41.482910    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:41.969190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:41.964576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:42.363675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:42.363636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:43.176676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:43.174115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:43.387226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:43.387183    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:45.076055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:45.076019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:46.901677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:46.901640    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:47.130045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:47.130000    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:48.889105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:48.889071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:49.288002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:49.287965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:50.806235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:50.806193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:51.031126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:51.031090    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:52.908005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:52.907972    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:53.141468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:53.141431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:54.818294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:54.818252    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:55.255233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.255139    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:29:55.255418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.255348    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy" containerID="cri-o://6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47" gracePeriod=30
Oct 31 04:29:55.340391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.340359    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:29:55.340973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.340957    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" containerName="maas-api"
Oct 31 04:29:55.341044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.340974    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" containerName="maas-api"
Oct 31 04:29:55.341098 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.341088    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccbec93-da10-49ec-9e23-6fb2f9f1aaf6" containerName="maas-api"
Oct 31 04:29:55.344471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.344448    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.406085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.406047    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:29:55.482326 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482290    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482410    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482442    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482536    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482629    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482676    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482734    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2zg\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482803 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482786    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.482884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.482864    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584449    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584529    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584551    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584582    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584634    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584684    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584713    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2zg\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584754    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584783    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.584955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584902    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.585173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.584962    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.585271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.585252    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.585410 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.585392    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.585484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.585446    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.587015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.586989    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.587147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.587129    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.609707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.609677    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.609986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.609964    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2zg\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\") pod \"e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") " pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.664751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.664718    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:55.988512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.988433    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:29:55.992093 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:29:55.992059    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87551beb_c484_4cf8_a770_5d9642f6d928.slice/crio-62bb92364daefa46425280776353c8b8f4a5c2031757aa77acee244e11ed81ca WatchSource:0}: Error finding container 62bb92364daefa46425280776353c8b8f4a5c2031757aa77acee244e11ed81ca: Status 404 returned error can't find the container with id 62bb92364daefa46425280776353c8b8f4a5c2031757aa77acee244e11ed81ca
Oct 31 04:29:55.994667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.994627    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:29:55.994791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.994720    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:29:55.994791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:55.994762    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:29:56.793049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:56.793018    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" event={"ID":"87551beb-c484-4cf8-a770-5d9642f6d928","Type":"ContainerStarted","Data":"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"}
Oct 31 04:29:56.793215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:56.793075    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" event={"ID":"87551beb-c484-4cf8-a770-5d9642f6d928","Type":"ContainerStarted","Data":"62bb92364daefa46425280776353c8b8f4a5c2031757aa77acee244e11ed81ca"}
Oct 31 04:29:56.855362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:56.855303    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podStartSLOduration=1.8552883329999998 podStartE2EDuration="1.855288333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:29:56.853559679 +0000 UTC m=+1492.886559380" watchObservedRunningTime="2026-10-31 04:29:56.855288333 +0000 UTC m=+1492.888288012"
Oct 31 04:29:56.966216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:56.966181    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:57.177827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:57.177745    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:57.665114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:57.665067    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:29:57.666634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:57.666607    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:29:57.666728 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:57.666671    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:29:57.754202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:57.754125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:58.665751 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:58.665705    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:29:58.666146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:58.665770    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:29:58.793708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:58.793673    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:29:58.793912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:58.793892    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" podUID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" containerName="maas-api" containerID="cri-o://c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912" gracePeriod=30
Oct 31 04:29:58.967845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:58.967755    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:59.202498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.202126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:59.206708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.206679    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:29:59.271364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.271279    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\") pod \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") "
Oct 31 04:29:59.271364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.271342    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\") pod \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") "
Oct 31 04:29:59.271594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.271491    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\" (UniqueName: \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\") pod \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") "
Oct 31 04:29:59.271594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.271549    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\") pod \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\" (UID: \"1ce5b5cd-695f-41fb-96e2-ab561cc93aec\") "
Oct 31 04:29:59.272296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.272267    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca" pod "1ce5b5cd-695f-41fb-96e2-ab561cc93aec" (UID: "1ce5b5cd-695f-41fb-96e2-ab561cc93aec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:29:59.275200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.275107    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls" pod "1ce5b5cd-695f-41fb-96e2-ab561cc93aec" (UID: "1ce5b5cd-695f-41fb-96e2-ab561cc93aec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:29:59.275670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.275648    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs" pod "1ce5b5cd-695f-41fb-96e2-ab561cc93aec" (UID: "1ce5b5cd-695f-41fb-96e2-ab561cc93aec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:29:59.275871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.275844    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj" pod "1ce5b5cd-695f-41fb-96e2-ab561cc93aec" (UID: "1ce5b5cd-695f-41fb-96e2-ab561cc93aec"). InnerVolumeSpecName "kube-api-access-mbbhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:29:59.373694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.373658    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:59.373694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.373688    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbhj\" (UniqueName: \"kubernetes.io/projected/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-kube-api-access-mbbhj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:59.373694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.373701    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:59.373964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.373711    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce5b5cd-695f-41fb-96e2-ab561cc93aec-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:29:59.432254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.428741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:59.665487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.665456    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:29:59.665689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.665512    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:29:59.816916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.816879    6931 generic.go:356] "Generic (PLEG): container finished" podID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" containerID="c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912" exitCode=0
Oct 31 04:29:59.817352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.817004    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" event={"ID":"1ce5b5cd-695f-41fb-96e2-ab561cc93aec","Type":"ContainerDied","Data":"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"}
Oct 31 04:29:59.817352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.817046    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb" event={"ID":"1ce5b5cd-695f-41fb-96e2-ab561cc93aec","Type":"ContainerDied","Data":"be5b79248a52c04cc9497cd5515f6596c17e3df7435129f935d95556f3ca46be"}
Oct 31 04:29:59.817352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.817070    6931 scope.go:122] "RemoveContainer" containerID="c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"
Oct 31 04:29:59.817352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.817218    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"
Oct 31 04:29:59.830870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.829105    6931 scope.go:122] "RemoveContainer" containerID="c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"
Oct 31 04:29:59.843178 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:29:59.834402    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912\": container with ID starting with c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912 not found: ID does not exist" containerID="c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"
Oct 31 04:29:59.843178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.834447    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912"} err="failed to get container status \"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912\": rpc error: code = NotFound desc = could not find container \"c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912\": container with ID starting with c94aa0bd6319916513f00e694b94ff75857646f2d38509513a6d3631afe23912 not found: ID does not exist"
Oct 31 04:29:59.875301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.875266    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:29:59.879188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:29:59.879128    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24cd29-d564b88dc-gqsjb"]
Oct 31 04:30:00.124183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.124130    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" path="/var/lib/kubelet/pods/1ce5b5cd-695f-41fb-96e2-ab561cc93aec/volumes"
Oct 31 04:30:00.177779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.177622    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"]
Oct 31 04:30:00.178832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.178686    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" containerName="maas-api"
Oct 31 04:30:00.179054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.179033    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" containerName="maas-api"
Oct 31 04:30:00.179351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.179335    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce5b5cd-695f-41fb-96e2-ab561cc93aec" containerName="maas-api"
Oct 31 04:30:00.207183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.190444    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.207183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.205714    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:00.221180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.212120    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.221180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.216425    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"]
Oct 31 04:30:00.221180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.220024    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:00.222610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.222285    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:00.223570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.223548    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:30:00.223717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.223598    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\""
Oct 31 04:30:00.268513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.265268    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:00.268513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.265393    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.268513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.268141    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:00.286116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286086    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.286290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286121    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.286290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286217    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.286429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286302    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcjw\" (UniqueName: \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.286525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286504    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.286585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286567    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.286645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.286595    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25gm\" (UniqueName: \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388109    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcjw\" (UniqueName: \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388355    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388417    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388441    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25gm\" (UniqueName: \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388488    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388513    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.388551    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.389177    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.389425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.389379    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.390022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.389780    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.391949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.391929    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.414543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.414502    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcjw\" (UniqueName: \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\") pod \"maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.414721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.414683    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\") pod \"maas-api-revoke-sub-b6c162de-jmh69\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.415263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.415247    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25gm\" (UniqueName: \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\") pod \"maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.576106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.575888    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:00.590757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.590730    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:00.596291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.596267    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:30:00.666023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.665941    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:30:00.666023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.665993    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:30:00.670801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.670567    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:00.693733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693705    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.693836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693762    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.693901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693831    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.693901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693859    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.694001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693919    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.694001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.693964    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.694209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.694191    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.694289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.694274    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.694347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.694330    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\") pod \"41691951-56fe-4f9a-92ac-2c45175692d3\" (UID: \"41691951-56fe-4f9a-92ac-2c45175692d3\") "
Oct 31 04:30:00.695313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.695282    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:00.695407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.695292    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:00.695498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.695479    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:00.696884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.696856    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:00.697042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.697021    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:00.700445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.700399    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "kube-api-access-g9slc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:00.700537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.700498    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:00.700606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.700581    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:00.701647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.701602    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo" pod "41691951-56fe-4f9a-92ac-2c45175692d3" (UID: "41691951-56fe-4f9a-92ac-2c45175692d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:30:00.797305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797269    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41691951-56fe-4f9a-92ac-2c45175692d3-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797306    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797320    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797333    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797347    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41691951-56fe-4f9a-92ac-2c45175692d3-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797360    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797374    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797387    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9slc\" (UniqueName: \"kubernetes.io/projected/41691951-56fe-4f9a-92ac-2c45175692d3-kube-api-access-g9slc\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.797479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.797401    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41691951-56fe-4f9a-92ac-2c45175692d3-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:00.851875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.851829    6931 generic.go:356] "Generic (PLEG): container finished" podID="41691951-56fe-4f9a-92ac-2c45175692d3" containerID="6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47" exitCode=0
Oct 31 04:30:00.852404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.851964    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" event={"ID":"41691951-56fe-4f9a-92ac-2c45175692d3","Type":"ContainerDied","Data":"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"}
Oct 31 04:30:00.852404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.852018    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf" event={"ID":"41691951-56fe-4f9a-92ac-2c45175692d3","Type":"ContainerDied","Data":"8f3e2a751d55f02c1ee7596e922afc71c1f3b99bc0677872580ba2ad8716b365"}
Oct 31 04:30:00.852404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.852053    6931 scope.go:122] "RemoveContainer" containerID="6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"
Oct 31 04:30:00.852404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.852195    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"
Oct 31 04:30:00.870648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.870612    6931 scope.go:122] "RemoveContainer" containerID="6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"
Oct 31 04:30:00.871059 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:00.870980    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47\": container with ID starting with 6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47 not found: ID does not exist" containerID="6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"
Oct 31 04:30:00.871183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.871073    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47"} err="failed to get container status \"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47\": rpc error: code = NotFound desc = could not find container \"6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47\": container with ID starting with 6fce7c977b9ac01b60958959d03bcb1b34608a5efe691ee80c88877b208d8a47 not found: ID does not exist"
Oct 31 04:30:00.924827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.924744    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"]
Oct 31 04:30:00.957660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.957621    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:30:00.964394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.964368    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-rmcgf"]
Oct 31 04:30:00.966275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:00.966015    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:01.019233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.018871    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:01.022104 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:30:01.022076    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21006aa4_c15d_4743_8eb0_ec689e9a274e.slice/crio-e49b8b06e4eb7d26e20599afdb558a4f32849fbb66bea3e9126e6d518c07358c WatchSource:0}: Error finding container e49b8b06e4eb7d26e20599afdb558a4f32849fbb66bea3e9126e6d518c07358c: Status 404 returned error can't find the container with id e49b8b06e4eb7d26e20599afdb558a4f32849fbb66bea3e9126e6d518c07358c
Oct 31 04:30:01.665716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.665684    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:30:01.665901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.665742    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:30:01.866228 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.866189    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerStarted","Data":"79b4713f3c3b850e5363e91b7d959fadf343024344bf6e1ed1cb8051bcbcd07e"}
Oct 31 04:30:01.868454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.868430    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69" event={"ID":"39212ede-3fbc-4c7c-a83b-23c9334064e1","Type":"ContainerStarted","Data":"9632c3d3bf34f15cdb78d71b26c7eb2c375cc3d2f8658ee2c093e4b5c826274d"}
Oct 31 04:30:01.869631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:01.869608    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerStarted","Data":"e49b8b06e4eb7d26e20599afdb558a4f32849fbb66bea3e9126e6d518c07358c"}
Oct 31 04:30:02.123234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:02.123198    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" path="/var/lib/kubelet/pods/41691951-56fe-4f9a-92ac-2c45175692d3/volumes"
Oct 31 04:30:02.635214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:02.635177    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:02.671191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:02.665674    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 04:30:02.671191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:02.665737    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 04:30:03.697589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:03.694495    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:30:03.697589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:03.696238    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:30:03.698373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:03.698034    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:30:04.556347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:04.556245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:04.824523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:04.824491    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:05.054054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.054023    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:05.626889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.626843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:05.907226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.907098    6931 generic.go:356] "Generic (PLEG): container finished" podID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerID="0ae06d4bbc863a498913eb242c1648b950efb6d1c8d932536a97b7c594e875ff" exitCode=60
Oct 31 04:30:05.907226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.907185    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerDied","Data":"0ae06d4bbc863a498913eb242c1648b950efb6d1c8d932536a97b7c594e875ff"}
Oct 31 04:30:05.907769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.907498    6931 scope.go:122] "RemoveContainer" containerID="0ae06d4bbc863a498913eb242c1648b950efb6d1c8d932536a97b7c594e875ff"
Oct 31 04:30:05.909533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.909507    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerStarted","Data":"084a7e1f6cf05fe7322d46dc040ae47cc24a0f87487bdb4019c0b172b75af701"}
Oct 31 04:30:05.911141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.911110    6931 generic.go:356] "Generic (PLEG): container finished" podID="39212ede-3fbc-4c7c-a83b-23c9334064e1" containerID="ec8c71a5134ee4f7e9fcebca14b8118dadb47739d8afcbf04f6b503bb9a34327" exitCode=0
Oct 31 04:30:05.911273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:05.911198    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69" event={"ID":"39212ede-3fbc-4c7c-a83b-23c9334064e1","Type":"ContainerDied","Data":"ec8c71a5134ee4f7e9fcebca14b8118dadb47739d8afcbf04f6b503bb9a34327"}
Oct 31 04:30:06.013304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.013266    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:06.043119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.043050    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" podStartSLOduration=2.1876002 podStartE2EDuration="6.043030291s" totalImagesPullingTime="3.855430091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:05.999452834 +0000 UTC m=+1502.032452533" watchObservedRunningTime="2026-10-31 04:30:06.043030291 +0000 UTC m=+1502.076029971"
Oct 31 04:30:06.430201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.427683    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:06.921846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.921265    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerStarted","Data":"180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"}
Oct 31 04:30:06.923204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.923139    6931 generic.go:356] "Generic (PLEG): container finished" podID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerID="084a7e1f6cf05fe7322d46dc040ae47cc24a0f87487bdb4019c0b172b75af701" exitCode=60
Oct 31 04:30:06.923344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.923214    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerDied","Data":"084a7e1f6cf05fe7322d46dc040ae47cc24a0f87487bdb4019c0b172b75af701"}
Oct 31 04:30:06.923539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.923518    6931 scope.go:122] "RemoveContainer" containerID="084a7e1f6cf05fe7322d46dc040ae47cc24a0f87487bdb4019c0b172b75af701"
Oct 31 04:30:06.966971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:06.966896    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" podStartSLOduration=3.16791883 podStartE2EDuration="6.966882073s" totalImagesPullingTime="3.798963243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:06.966499334 +0000 UTC m=+1502.999499010" watchObservedRunningTime="2026-10-31 04:30:06.966882073 +0000 UTC m=+1502.999881751"
Oct 31 04:30:07.219652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.219583    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:07.300117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.300078    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\") pod \"39212ede-3fbc-4c7c-a83b-23c9334064e1\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") "
Oct 31 04:30:07.300308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.300264    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\") pod \"39212ede-3fbc-4c7c-a83b-23c9334064e1\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") "
Oct 31 04:30:07.300404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.300381    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\") pod \"39212ede-3fbc-4c7c-a83b-23c9334064e1\" (UID: \"39212ede-3fbc-4c7c-a83b-23c9334064e1\") "
Oct 31 04:30:07.300534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.300506    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca" pod "39212ede-3fbc-4c7c-a83b-23c9334064e1" (UID: "39212ede-3fbc-4c7c-a83b-23c9334064e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:07.300823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.300798    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-openshift-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:07.302573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.302548    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q" pod "39212ede-3fbc-4c7c-a83b-23c9334064e1" (UID: "39212ede-3fbc-4c7c-a83b-23c9334064e1"). InnerVolumeSpecName "kube-api-access-fqp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:07.302948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.302922    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca" pod "39212ede-3fbc-4c7c-a83b-23c9334064e1" (UID: "39212ede-3fbc-4c7c-a83b-23c9334064e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:30:07.404845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.402906    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39212ede-3fbc-4c7c-a83b-23c9334064e1-maas-api-xks-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:07.404845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.402955    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqp2q\" (UniqueName: \"kubernetes.io/projected/39212ede-3fbc-4c7c-a83b-23c9334064e1-kube-api-access-fqp2q\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:07.511198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.511087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:07.722602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.722570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:07.932600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.932572    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerStarted","Data":"3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"}
Oct 31 04:30:07.934082 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.934062    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"
Oct 31 04:30:07.934205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.934110    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69" event={"ID":"39212ede-3fbc-4c7c-a83b-23c9334064e1","Type":"ContainerDied","Data":"9632c3d3bf34f15cdb78d71b26c7eb2c375cc3d2f8658ee2c093e4b5c826274d"}
Oct 31 04:30:07.934205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.934190    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9632c3d3bf34f15cdb78d71b26c7eb2c375cc3d2f8658ee2c093e4b5c826274d"
Oct 31 04:30:07.939981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.939955    6931 generic.go:356] "Generic (PLEG): container finished" podID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerID="180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af" exitCode=60
Oct 31 04:30:07.940996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.940028    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerDied","Data":"180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"}
Oct 31 04:30:07.940996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.940077    6931 scope.go:122] "RemoveContainer" containerID="0ae06d4bbc863a498913eb242c1648b950efb6d1c8d932536a97b7c594e875ff"
Oct 31 04:30:07.940996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:07.940491    6931 scope.go:122] "RemoveContainer" containerID="180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"
Oct 31 04:30:07.940996 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:07.940790    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt_odh-ai-gateway-infra(21006aa4-c15d-4743-8eb0-ec689e9a274e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e"
Oct 31 04:30:08.018209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.016069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:08.226971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.226510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:08.947742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.947712    6931 scope.go:122] "RemoveContainer" containerID="180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"
Oct 31 04:30:08.948146 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:08.948063    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt_odh-ai-gateway-infra(21006aa4-c15d-4743-8eb0-ec689e9a274e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e"
Oct 31 04:30:08.949416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.949387    6931 generic.go:356] "Generic (PLEG): container finished" podID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerID="3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3" exitCode=60
Oct 31 04:30:08.949541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.949418    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerDied","Data":"3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"}
Oct 31 04:30:08.949541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.949501    6931 scope.go:122] "RemoveContainer" containerID="084a7e1f6cf05fe7322d46dc040ae47cc24a0f87487bdb4019c0b172b75af701"
Oct 31 04:30:08.949826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:08.949803    6931 scope.go:122] "RemoveContainer" containerID="3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"
Oct 31 04:30:08.950069 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:08.950044    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr_odh-ai-gateway-infra(6e3deca7-147e-4e2e-9fff-1149acdfc88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f"
Oct 31 04:30:09.272300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:09.272199    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:09.509366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:09.509322    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:09.723735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:09.723701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:09.959118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:09.959093    6931 scope.go:122] "RemoveContainer" containerID="3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"
Oct 31 04:30:09.959539 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:09.959384    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr_odh-ai-gateway-infra(6e3deca7-147e-4e2e-9fff-1149acdfc88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f"
Oct 31 04:30:10.146231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:10.144782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:10.628552 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:10.627736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:10.779128 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:10.777829    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:30:11.518442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:11.518406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:11.769235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:11.759147    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:12.585018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:12.584976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:14.621112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:14.621075    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:14.835057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:14.835015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:15.050771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:15.050731    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:16.653403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:16.653369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:17.349727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:17.349685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:18.657836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:18.657133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:19.128125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:19.128090    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:20.508653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:20.508618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:21.115245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:21.115211    6931 scope.go:122] "RemoveContainer" containerID="3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"
Oct 31 04:30:21.115425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:21.115320    6931 scope.go:122] "RemoveContainer" containerID="180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"
Oct 31 04:30:21.437532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:21.437113    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:30:21.437532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:21.437331    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy" containerID="cri-o://f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37" gracePeriod=30
Oct 31 04:30:22.083732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:22.083694    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerStarted","Data":"61d8a3ce7c87fd6dd16adde33b08e5464a97cce968f0c70b25860cadcfd6c039"}
Oct 31 04:30:22.086447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:22.086222    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerStarted","Data":"bcdde7087ce1ab671013fac91b01561bf591714ab8ec3f47bfd5d486fb9f17db"}
Oct 31 04:30:22.403909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:22.403812    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:23.094011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.093976    6931 generic.go:356] "Generic (PLEG): container finished" podID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerID="bcdde7087ce1ab671013fac91b01561bf591714ab8ec3f47bfd5d486fb9f17db" exitCode=60
Oct 31 04:30:23.094535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.094048    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerDied","Data":"bcdde7087ce1ab671013fac91b01561bf591714ab8ec3f47bfd5d486fb9f17db"}
Oct 31 04:30:23.094535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.094113    6931 scope.go:122] "RemoveContainer" containerID="3b47463d4e9429b70660dcf9d43ddfb98a075894d51b3052ec83c9209ef64ae3"
Oct 31 04:30:23.094535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.094466    6931 scope.go:122] "RemoveContainer" containerID="bcdde7087ce1ab671013fac91b01561bf591714ab8ec3f47bfd5d486fb9f17db"
Oct 31 04:30:23.094817 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:23.094792    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr_odh-ai-gateway-infra(6e3deca7-147e-4e2e-9fff-1149acdfc88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f"
Oct 31 04:30:23.096295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.096275    6931 generic.go:356] "Generic (PLEG): container finished" podID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerID="61d8a3ce7c87fd6dd16adde33b08e5464a97cce968f0c70b25860cadcfd6c039" exitCode=60
Oct 31 04:30:23.096430 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.096326    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerDied","Data":"61d8a3ce7c87fd6dd16adde33b08e5464a97cce968f0c70b25860cadcfd6c039"}
Oct 31 04:30:23.096646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.096632    6931 scope.go:122] "RemoveContainer" containerID="61d8a3ce7c87fd6dd16adde33b08e5464a97cce968f0c70b25860cadcfd6c039"
Oct 31 04:30:23.096885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:23.096869    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt_odh-ai-gateway-infra(21006aa4-c15d-4743-8eb0-ec689e9a274e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e"
Oct 31 04:30:23.106593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.106572    6931 scope.go:122] "RemoveContainer" containerID="180e8024fcd764167f343b1ec27c864af1e7c2f300e5cd88f58acaf59483c2af"
Oct 31 04:30:23.166200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.166145    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:23.172953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:23.172921    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:24.323215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.314282    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:24.374762 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.374737    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:24.378485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.378463    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:24.404564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.404533    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\") pod \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") "
Oct 31 04:30:24.404729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.404579    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\") pod \"21006aa4-c15d-4743-8eb0-ec689e9a274e\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") "
Oct 31 04:30:24.404729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.404639    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\" (UniqueName: \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\") pod \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\" (UID: \"6e3deca7-147e-4e2e-9fff-1149acdfc88f\") "
Oct 31 04:30:24.404862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.404769    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\" (UniqueName: \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\") pod \"21006aa4-c15d-4743-8eb0-ec689e9a274e\" (UID: \"21006aa4-c15d-4743-8eb0-ec689e9a274e\") "
Oct 31 04:30:24.405075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.405043    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca" pod "6e3deca7-147e-4e2e-9fff-1149acdfc88f" (UID: "6e3deca7-147e-4e2e-9fff-1149acdfc88f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:24.405195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.405095    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca" pod "21006aa4-c15d-4743-8eb0-ec689e9a274e" (UID: "21006aa4-c15d-4743-8eb0-ec689e9a274e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:24.405405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.405384    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21006aa4-c15d-4743-8eb0-ec689e9a274e-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:24.405486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.405407    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3deca7-147e-4e2e-9fff-1149acdfc88f-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:24.406867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.406845    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw" pod "6e3deca7-147e-4e2e-9fff-1149acdfc88f" (UID: "6e3deca7-147e-4e2e-9fff-1149acdfc88f"). InnerVolumeSpecName "kube-api-access-7kcjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:24.406946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.406877    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm" pod "21006aa4-c15d-4743-8eb0-ec689e9a274e" (UID: "21006aa4-c15d-4743-8eb0-ec689e9a274e"). InnerVolumeSpecName "kube-api-access-n25gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:24.506506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.506475    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kcjw\" (UniqueName: \"kubernetes.io/projected/6e3deca7-147e-4e2e-9fff-1149acdfc88f-kube-api-access-7kcjw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:24.506506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:24.506504    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25gm\" (UniqueName: \"kubernetes.io/projected/21006aa4-c15d-4743-8eb0-ec689e9a274e-kube-api-access-n25gm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:25.113729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.113699    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"
Oct 31 04:30:25.113916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.113734    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr" event={"ID":"6e3deca7-147e-4e2e-9fff-1149acdfc88f","Type":"ContainerDied","Data":"79b4713f3c3b850e5363e91b7d959fadf343024344bf6e1ed1cb8051bcbcd07e"}
Oct 31 04:30:25.113916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.113836    6931 scope.go:122] "RemoveContainer" containerID="bcdde7087ce1ab671013fac91b01561bf591714ab8ec3f47bfd5d486fb9f17db"
Oct 31 04:30:25.115462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.115441    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt" event={"ID":"21006aa4-c15d-4743-8eb0-ec689e9a274e","Type":"ContainerDied","Data":"e49b8b06e4eb7d26e20599afdb558a4f32849fbb66bea3e9126e6d518c07358c"}
Oct 31 04:30:25.115462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.115456    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"
Oct 31 04:30:25.124712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.124694    6931 scope.go:122] "RemoveContainer" containerID="61d8a3ce7c87fd6dd16adde33b08e5464a97cce968f0c70b25860cadcfd6c039"
Oct 31 04:30:25.165697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.165663    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:25.177359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.177328    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-09a20a-29890350-j2clr"]
Oct 31 04:30:25.207736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.207702    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:25.222086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.222060    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-439989-29890350-f74vt"]
Oct 31 04:30:25.802563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:25.802524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:26.038821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.038784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:26.123611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.123536    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" path="/var/lib/kubelet/pods/21006aa4-c15d-4743-8eb0-ec689e9a274e/volumes"
Oct 31 04:30:26.124127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.124107    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" path="/var/lib/kubelet/pods/6e3deca7-147e-4e2e-9fff-1149acdfc88f/volumes"
Oct 31 04:30:26.707994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.707936    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:30:26.738124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738094    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738135    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738245    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738324    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738436    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738467    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738507    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738558    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:26.738592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738570    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738750 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738655    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\") pod \"711397bd-dfac-4257-9fcc-54cfa1daba92\" (UID: \"711397bd-dfac-4257-9fcc-54cfa1daba92\") "
Oct 31 04:30:26.738831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.738805    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:26.739308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739225    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:26.739507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739484    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/711397bd-dfac-4257-9fcc-54cfa1daba92-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.739601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739513    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.739601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739526    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.739712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739584    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:26.739866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.739848    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:26.740678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.740651    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:30:26.740872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.740850    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:26.740922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.740883    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "kube-api-access-2wfh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:26.741065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.741053    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy" pod "711397bd-dfac-4257-9fcc-54cfa1daba92" (UID: "711397bd-dfac-4257-9fcc-54cfa1daba92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:26.841405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841377    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.841405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841405    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfh4\" (UniqueName: \"kubernetes.io/projected/711397bd-dfac-4257-9fcc-54cfa1daba92-kube-api-access-2wfh4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.841793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841415    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.841793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841429    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.841793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841438    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.841793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.841452    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/711397bd-dfac-4257-9fcc-54cfa1daba92-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:26.857682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:26.857646    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:27.132044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.132003    6931 generic.go:356] "Generic (PLEG): container finished" podID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerID="f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37" exitCode=0
Oct 31 04:30:27.132247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.132063    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"
Oct 31 04:30:27.132247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.132088    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" event={"ID":"711397bd-dfac-4257-9fcc-54cfa1daba92","Type":"ContainerDied","Data":"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"}
Oct 31 04:30:27.132247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.132141    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g" event={"ID":"711397bd-dfac-4257-9fcc-54cfa1daba92","Type":"ContainerDied","Data":"20431ee0023d39ec3cadf0d317b58be1b992d5b851450ab71ec997085459ea75"}
Oct 31 04:30:27.132247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.132180    6931 scope.go:122] "RemoveContainer" containerID="f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"
Oct 31 04:30:27.142442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.142423    6931 scope.go:122] "RemoveContainer" containerID="f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"
Oct 31 04:30:27.142700 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:27.142680    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37\": container with ID starting with f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37 not found: ID does not exist" containerID="f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"
Oct 31 04:30:27.142756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.142708    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37"} err="failed to get container status \"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37\": rpc error: code = NotFound desc = could not find container \"f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37\": container with ID starting with f7c01c4ad99ab827bd7f8a7c387c6f4acf00affe1344a130f77bf45e2ee79f37 not found: ID does not exist"
Oct 31 04:30:27.184214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.184184    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:30:27.195808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:27.195781    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-rnr8g"]
Oct 31 04:30:28.124002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:28.123970    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" path="/var/lib/kubelet/pods/711397bd-dfac-4257-9fcc-54cfa1daba92/volumes"
Oct 31 04:30:28.823756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:28.822480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:30.465439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:30.465410    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:30:30.465883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:30.465656    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy" containerID="cri-o://445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee" gracePeriod=30
Oct 31 04:30:30.723461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:30.722520    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:30.938780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:30.938740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:31.816387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:31.816332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:32.026069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:32.026026    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:32.565944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:32.565907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:32.773316 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:32.773281    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:32.995138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:32.995047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:33.769920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:33.769887    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:34.604496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:34.604460    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:35.743746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.743684    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:30:35.869586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869551    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869589    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869629    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869683    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869725    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869754    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.869974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869848    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:35.870035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.869970    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.870091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870042    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.870144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870092    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:35.870144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870121    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\") pod \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\" (UID: \"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99\") "
Oct 31 04:30:35.870302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870137    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:35.870302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870199    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:35.870302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870260    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:35.870643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870621    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.870717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870647    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.870717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870661    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.870717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870675    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.870717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.870690    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.872114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.872093    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:35.872215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.872128    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:30:35.872272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.872234    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:30:35.872307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.872281    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv" pod "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" (UID: "2a48553f-8d2f-48c6-b2f1-3d3e359c8d99"). InnerVolumeSpecName "kube-api-access-nx6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:35.972214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.972142    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.972214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.972209    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.972214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.972218    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:35.972456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:35.972227    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6jv\" (UniqueName: \"kubernetes.io/projected/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99-kube-api-access-nx6jv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:30:36.196572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.196535    6931 generic.go:356] "Generic (PLEG): container finished" podID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerID="445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee" exitCode=0
Oct 31 04:30:36.196776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.196590    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"
Oct 31 04:30:36.196776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.196627    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" event={"ID":"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99","Type":"ContainerDied","Data":"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"}
Oct 31 04:30:36.196776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.196689    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj" event={"ID":"2a48553f-8d2f-48c6-b2f1-3d3e359c8d99","Type":"ContainerDied","Data":"6683ce04c2f6c9d4d5dddcf3a90b1eafe2b4dc3e419ea97bbfd7c5c3c559ebaa"}
Oct 31 04:30:36.196776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.196718    6931 scope.go:122] "RemoveContainer" containerID="445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"
Oct 31 04:30:36.206110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.206083    6931 scope.go:122] "RemoveContainer" containerID="445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"
Oct 31 04:30:36.207511 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:36.206985    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee\": container with ID starting with 445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee not found: ID does not exist" containerID="445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"
Oct 31 04:30:36.207511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.207033    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee"} err="failed to get container status \"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee\": rpc error: code = NotFound desc = could not find container \"445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee\": container with ID starting with 445f988a0c61e92128f9457c5494a9bd735d0e42d209ccf079bd3000db973dee not found: ID does not exist"
Oct 31 04:30:36.248238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.248192    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:30:36.260341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.260299    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-g99lj"]
Oct 31 04:30:36.510487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:36.510388    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:37.176412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:37.176372    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:37.390229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:37.390190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:38.128791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:38.128740    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" path="/var/lib/kubelet/pods/2a48553f-8d2f-48c6-b2f1-3d3e359c8d99/volumes"
Oct 31 04:30:38.541753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:38.541719    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:39.898670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:39.898634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:40.355101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:40.355061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:40.566265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:40.566222    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:40.829086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:40.828975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:40.832749 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:30:40.832644    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:30:42.687706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:42.687660    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:44.576638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:44.576544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:46.510292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:46.510249    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:46.724141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:46.724105    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:48.352637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:48.352601    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:50.103597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:50.103559    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:52.038958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:52.038912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:53.584225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:53.584149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:54.425572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:54.425536    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:56.176337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:56.176290    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:57.069438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:57.069399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:57.883305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:57.883266    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:58.105603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:58.105569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:58.318952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:58.318914    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:58.724350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:58.724262    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:58.933069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:58.933015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:59.959752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:30:59.959392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:01.785791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:01.785756    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:02.001857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:02.001817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:02.396935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:02.396897    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:02.604446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:02.604405    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:02.835922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:02.835883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:03.066778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:03.066717    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:04.575452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:04.575406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:04.796794 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:04.796752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:05.718298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:05.718262    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:06.636589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:06.636556    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:08.516687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:08.516649    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:09.378782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:09.378748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:09.587978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:09.587938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:10.568882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:10.568846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:10.761268 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:31:10.761227    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:31:10.788554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:10.788493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:12.705292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:12.705219    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:13.568988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:13.568953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:14.700002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:14.699958    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:16.601322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:16.601275    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:18.720212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:18.719690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:20.663208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663172    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:31:20.663704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663691    6931 cpu_manager.go:414] "containerMap: removing container" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.663748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663705    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.663748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663726    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663731    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663740    6931 cpu_manager.go:414] "containerMap: removing container" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy"
Oct 31 04:31:20.663748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663745    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663754    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663758    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663767    6931 cpu_manager.go:414] "containerMap: removing container" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663771    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663781    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663786    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663802    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663806    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663816    6931 cpu_manager.go:414] "containerMap: removing container" podUID="39212ede-3fbc-4c7c-a83b-23c9334064e1" containerName="revoke-keys"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663821    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39212ede-3fbc-4c7c-a83b-23c9334064e1" containerName="revoke-keys"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663843    6931 cpu_manager.go:414] "containerMap: removing container" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.663903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663848    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663929    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663938    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663946    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663954    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="711397bd-dfac-4257-9fcc-54cfa1daba92" containerName="istio-proxy"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663960    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a48553f-8d2f-48c6-b2f1-3d3e359c8d99" containerName="istio-proxy"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663965    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="41691951-56fe-4f9a-92ac-2c45175692d3" containerName="istio-proxy"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663971    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663980    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3deca7-147e-4e2e-9fff-1149acdfc88f" containerName="cleanup"
Oct 31 04:31:20.664291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.663987    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="39212ede-3fbc-4c7c-a83b-23c9334064e1" containerName="revoke-keys"
Oct 31 04:31:20.668420 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.668400    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.686604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.686574    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e6a0de-serving-cert\""
Oct 31 04:31:20.702830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.702792    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:31:20.715376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.715340    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.715661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.715637    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.715788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.715672    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.715788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.715706    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jcj\" (UniqueName: \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.816784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.816747    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.816959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.816829    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.816959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.816859    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.816959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.816896    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jcj\" (UniqueName: \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.816959 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:31:20.816909    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-serving-cert: secret "maas-api-e2e-praxis-b-e6a0de-serving-cert" not found
Oct 31 04:31:20.817140 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:31:20.816991    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls podName:ee892a00-0f88-40dd-b431-9d441000a58c nodeName:}" failed. No retries permitted until 2026-10-31 04:31:21.316970368 +0000 UTC m=+1577.349970036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls") pod "maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" (UID: "ee892a00-0f88-40dd-b431-9d441000a58c") : secret "maas-api-e2e-praxis-b-e6a0de-serving-cert" not found
Oct 31 04:31:20.817618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.817595    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.819581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.819537    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.850689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.850466    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jcj\" (UniqueName: \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:20.868976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:20.868943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:21.082858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.082819    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:21.324843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.324809    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:21.327247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.327223    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:21.582321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.582286    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:21.900192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.900083    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:31:21.904377 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:31:21.904346    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee892a00_0f88_40dd_b431_9d441000a58c.slice/crio-9523cd471ebdc935bf988d636c1445b96b1fe36d38f195d1598b8d34499a3d9e WatchSource:0}: Error finding container 9523cd471ebdc935bf988d636c1445b96b1fe36d38f195d1598b8d34499a3d9e: Status 404 returned error can't find the container with id 9523cd471ebdc935bf988d636c1445b96b1fe36d38f195d1598b8d34499a3d9e
Oct 31 04:31:21.912380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.912349    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:31:21.913009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.912991    6931 cpu_manager.go:414] "containerMap: removing container" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:21.913095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.913010    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:21.913246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.913230    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="21006aa4-c15d-4743-8eb0-ec689e9a274e" containerName="cleanup"
Oct 31 04:31:21.918004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.917984    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.939268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.939236    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:31:21.942669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.942642    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h7j\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.942775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.942685    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.942816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.942794    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.942950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.942928    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.943014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.942970    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.943014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.943000    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.943111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.943092    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.943262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.943243    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:21.943321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:21.943280    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.050820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.049756    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051068    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051263    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051390    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051506    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051649    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.051903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.051887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.052019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.052005    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.052266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.052249    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h7j\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.053191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.053148    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.053864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.053845    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.054186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.054151    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.070325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.070290    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.075589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.075534    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.076963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.076927    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.078727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.078674    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.080776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.080744    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h7j\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.093860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.093832    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.231930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.231852    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:22.569091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.569064    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:31:22.571932 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:31:22.571907    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef3cd76_b15a_4b81_8c64_e56b33633fa8.slice/crio-b098a29694c0cc4cf86d77c2603ab1acb1907b84b19e022b86ae3f93921c7890 WatchSource:0}: Error finding container b098a29694c0cc4cf86d77c2603ab1acb1907b84b19e022b86ae3f93921c7890: Status 404 returned error can't find the container with id b098a29694c0cc4cf86d77c2603ab1acb1907b84b19e022b86ae3f93921c7890
Oct 31 04:31:22.574373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.574338    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:22.574472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.574430    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:22.574472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.574461    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:22.704563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.704415    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" event={"ID":"ee892a00-0f88-40dd-b431-9d441000a58c","Type":"ContainerStarted","Data":"627679d705eccc6d1cef226eec7ceff3ec30b3a071b971d8ddd8e6cb3bbdcc6c"}
Oct 31 04:31:22.704744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.704474    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" event={"ID":"ee892a00-0f88-40dd-b431-9d441000a58c","Type":"ContainerStarted","Data":"9523cd471ebdc935bf988d636c1445b96b1fe36d38f195d1598b8d34499a3d9e"}
Oct 31 04:31:22.705570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.705115    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:22.707229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.706737    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" event={"ID":"0ef3cd76-b15a-4b81-8c64-e56b33633fa8","Type":"ContainerStarted","Data":"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"}
Oct 31 04:31:22.707229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.706789    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" event={"ID":"0ef3cd76-b15a-4b81-8c64-e56b33633fa8","Type":"ContainerStarted","Data":"b098a29694c0cc4cf86d77c2603ab1acb1907b84b19e022b86ae3f93921c7890"}
Oct 31 04:31:22.748546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.748493    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" podStartSLOduration=2.263055426 podStartE2EDuration="2.748479521s" totalImagesPullingTime="485.424095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:22.745777965 +0000 UTC m=+1578.778777662" watchObservedRunningTime="2026-10-31 04:31:22.748479521 +0000 UTC m=+1578.781479200"
Oct 31 04:31:22.791493 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:22.791441    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podStartSLOduration=1.791426552 podStartE2EDuration="1.791426552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:22.78992331 +0000 UTC m=+1578.822922989" watchObservedRunningTime="2026-10-31 04:31:22.791426552 +0000 UTC m=+1578.824426231"
Oct 31 04:31:23.007752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:23.007643    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:23.243641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:23.234590    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:23.243841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:23.243723    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:23.243841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:23.243779    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:24.251184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:24.248260    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:24.251184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:24.248329    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:24.903136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:24.903095    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:25.254053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:25.253964    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:25.254053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:25.254022    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:26.250181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:26.245818    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:26.250181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:26.245917    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:26.799785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:26.799672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:27.251614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:27.251532    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:27.251614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:27.251586    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:28.232574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:28.232536    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:28.233198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:28.233142    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:28.612472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:28.612331    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:28.719135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:28.719106    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:31:29.233292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:29.233253    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 04:31:29.233785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:29.233332    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 04:31:30.237564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:30.237526    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:30.238009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:30.237900    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:30.238767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:30.238749    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:31:30.497549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:30.497447    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:32.361821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:32.361783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:34.126467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:34.126430    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:36.001662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:36.001620    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:37.811893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:37.811859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:39.931768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:39.931715    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:41.050801 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:31:41.050732    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:31:42.402713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:42.402678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:43.262304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:43.262265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:44.420185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:44.420107    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:44.915550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:44.915513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:46.308680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:46.308640    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:46.837520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:46.837489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:47.464396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:47.464357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:47.672928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:47.672890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:48.420745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:48.420701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:48.639609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:48.639559    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:50.252772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:50.252738    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:51.192364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:51.192332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:51.419000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:51.418963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:52.208069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:52.208034    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:54.120317 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:54.120282    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:55.303245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.303208    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:55.641666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.641563    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:31:55.645952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.645927    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.671491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.671451    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:31:55.715375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715335    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715381    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715465    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715536    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715590    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715664    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7hn4\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715732    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715830    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.715894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.715861    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.817508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.817474    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.818852 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.818825    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.818995    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819028    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819048    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819084    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819141    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819180    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7hn4\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819213    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819211    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819432    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819498    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.819954 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.819933    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.820056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.820035    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.821309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.821286    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.821773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.821753    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.842853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.842819    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.843009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.842888    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7hn4\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\") pod \"e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") " pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:55.975970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.975752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:55.976731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:55.976440    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:56.483905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:56.483715    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:31:56.493309 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:31:56.493227    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f42f8b8_5ff2_4c78_99cb_fab4b0c0fa1d.slice/crio-16f8322236ed17b8a59dc3a1709f96511f816d92790858a93341d2fe39ba277b WatchSource:0}: Error finding container 16f8322236ed17b8a59dc3a1709f96511f816d92790858a93341d2fe39ba277b: Status 404 returned error can't find the container with id 16f8322236ed17b8a59dc3a1709f96511f816d92790858a93341d2fe39ba277b
Oct 31 04:31:56.495986 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:56.495940    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:56.496080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:56.496041    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:56.496120 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:56.496089    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:31:57.112181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.109361    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" event={"ID":"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d","Type":"ContainerStarted","Data":"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"}
Oct 31 04:31:57.112181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.109434    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" event={"ID":"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d","Type":"ContainerStarted","Data":"16f8322236ed17b8a59dc3a1709f96511f816d92790858a93341d2fe39ba277b"}
Oct 31 04:31:57.178194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.176079    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podStartSLOduration=2.176057811 podStartE2EDuration="2.176057811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:57.161966423 +0000 UTC m=+1613.194966131" watchObservedRunningTime="2026-10-31 04:31:57.176057811 +0000 UTC m=+1613.209057491"
Oct 31 04:31:57.638184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.637204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:57.983235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.982947    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:31:57.983235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.983187    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:31:57.983484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:57.983267    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:31:58.031181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.030300    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 31 04:31:58.086180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.085244    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:58.256235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.255877    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:58.280183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.277971    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:58.435181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.421489    6931 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:58.435181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.421794    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 31 04:31:58.435181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.424961    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:58.494208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.493312    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:58.981191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.979581    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:31:58.981191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:58.979646    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:31:59.085760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.085706    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:59.086527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.086507    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:31:59.087288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.087265    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:31:59.088499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.088207    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:31:59.189576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.189544    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 04:31:59.206765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.206735    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 04:31:59.643000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.642965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:59.978332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.978253    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:31:59.978332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:31:59.978312    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:32:00.977952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:00.977917    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:32:00.978416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:00.977976    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:32:01.562347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:01.561862    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:01.772371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:01.772333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:01.978188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:01.978076    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:32:01.978188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:01.978133    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:32:02.977897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:02.977859    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:32:02.978064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:02.977917    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:32:03.419661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:03.419626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:03.978337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:03.978302    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 04:32:03.978569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:03.978374    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 04:32:05.070764 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:05.070733    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:32:05.072028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:05.071995    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:32:05.073192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:05.073138    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:32:05.389584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:05.389485    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:06.978831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:06.978792    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:07.436139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:07.436096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:08.259926 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:08.259887    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:08.487745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:08.487629    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:09.056078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:09.056044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:09.277521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:09.277486    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:09.504116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:09.504015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:10.220311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:10.220277    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:10.555438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:10.555404    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:10.765343 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:10.765308    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:32:10.779833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:10.779800    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:12.756415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:12.756377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:13.043565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:13.043523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:13.257627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:13.257587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:13.819349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:13.819308    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:14.040045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:14.040003    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:14.264049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:14.263774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:15.094691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:15.094644    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:15.316916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:15.316878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:15.544090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:15.544053    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:15.753425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:15.753391    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:16.137222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:16.112107    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:16.364768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:16.364733    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:17.223486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:17.223449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:19.255112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:19.255068    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:19.496688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:19.496616    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:19.711193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:19.711142    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:20.668180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:20.651468    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:20.857954 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:20.857915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:21.580404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:21.580359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:22.269446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:22.269355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:22.584576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:22.584536    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:22.838915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:22.838831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:23.047869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:23.047829    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:23.271831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:23.271741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:24.295178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:24.294343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:24.585537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:24.585390    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:26.830417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:26.830385    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:32:26.830834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:26.830638    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy" containerID="cri-o://afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6" gracePeriod=30
Oct 31 04:32:26.956729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:26.956689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:27.753813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:27.753779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:27.961959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:27.961924    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:28.336876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:28.336846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:28.998005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:28.997964    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:29.540527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:29.540490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:29.784216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:29.784130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:30.000323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:30.000232    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:31.169108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:31.169073    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:31.375644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:31.375608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:32.149485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.149458    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:32:32.207893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.207729    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.207893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.207777    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.208527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208100    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:32:32.208638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208619    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.208736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208681    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.208736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208714    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.209047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208899    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.209047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.208940    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.209047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209030    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.209231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209173    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\") pod \"9d3ad619-d9db-4a40-8a33-04f861625790\" (UID: \"9d3ad619-d9db-4a40-8a33-04f861625790\") "
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209512    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209797    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209877    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.209912    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.210305    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:32:32.210868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.210517    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:32:32.211473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.211447    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:32.212529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.212490    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:32.213974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.213949    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "kube-api-access-bn74p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:32.218421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.218391    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy" pod "9d3ad619-d9db-4a40-8a33-04f861625790" (UID: "9d3ad619-d9db-4a40-8a33-04f861625790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:32:32.311703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311670    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311700    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn74p\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-kube-api-access-bn74p\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311709    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311717    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3ad619-d9db-4a40-8a33-04f861625790-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311728    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3ad619-d9db-4a40-8a33-04f861625790-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311736    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3ad619-d9db-4a40-8a33-04f861625790-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.311950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.311744    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3ad619-d9db-4a40-8a33-04f861625790-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:32.573576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.573542    6931 generic.go:356] "Generic (PLEG): container finished" podID="9d3ad619-d9db-4a40-8a33-04f861625790" containerID="afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6" exitCode=0
Oct 31 04:32:32.573761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.573591    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"
Oct 31 04:32:32.573761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.573599    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" event={"ID":"9d3ad619-d9db-4a40-8a33-04f861625790","Type":"ContainerDied","Data":"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"}
Oct 31 04:32:32.573761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.573645    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb" event={"ID":"9d3ad619-d9db-4a40-8a33-04f861625790","Type":"ContainerDied","Data":"207b7cdc14c9cc3693ddd69b63ea96a48bf93021182f1ee499280889fb123a06"}
Oct 31 04:32:32.573761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.573666    6931 scope.go:122] "RemoveContainer" containerID="afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"
Oct 31 04:32:32.584990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.584971    6931 scope.go:122] "RemoveContainer" containerID="afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"
Oct 31 04:32:32.585290 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:32.585270    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6\": container with ID starting with afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6 not found: ID does not exist" containerID="afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"
Oct 31 04:32:32.585369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.585299    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6"} err="failed to get container status \"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6\": rpc error: code = NotFound desc = could not find container \"afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6\": container with ID starting with afa4977fb45efb2c5a1d7f54f70bd9931107b55242f14e06a77dd38aa7713ae6 not found: ID does not exist"
Oct 31 04:32:32.625654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.625596    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:32:32.636963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:32.636933    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gnfkb"]
Oct 31 04:32:33.302509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:33.302474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:34.122515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:34.122480    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" path="/var/lib/kubelet/pods/9d3ad619-d9db-4a40-8a33-04f861625790/volumes"
Oct 31 04:32:34.709491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:34.709457    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:32:34.710405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:34.709721    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" podUID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" containerName="maas-api" containerID="cri-o://bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e" gracePeriod=30
Oct 31 04:32:34.937238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:34.937198    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:35.074657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.074628    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:32:35.160669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.160611    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\") pod \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") "
Oct 31 04:32:35.160864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.160741    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\" (UniqueName: \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\") pod \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") "
Oct 31 04:32:35.160930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.160898    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\") pod \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") "
Oct 31 04:32:35.161026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.161006    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\") pod \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\" (UID: \"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c\") "
Oct 31 04:32:35.161933 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.161904    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca" pod "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" (UID: "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:35.163512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.163441    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls" pod "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" (UID: "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:35.163512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.163447    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx" pod "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" (UID: "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c"). InnerVolumeSpecName "kube-api-access-24jxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:35.166756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.166724    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs" pod "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" (UID: "5ec7a65f-5ab6-4e44-b005-ad7ffae8943c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:35.263071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.263031    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:35.263071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.263066    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jxx\" (UniqueName: \"kubernetes.io/projected/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-kube-api-access-24jxx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:35.263071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.263079    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:35.263359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.263091    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:35.608223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.608180    6931 generic.go:356] "Generic (PLEG): container finished" podID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" containerID="bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e" exitCode=0
Oct 31 04:32:35.608409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.608250    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"
Oct 31 04:32:35.608409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.608262    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" event={"ID":"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c","Type":"ContainerDied","Data":"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"}
Oct 31 04:32:35.608409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.608316    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476" event={"ID":"5ec7a65f-5ab6-4e44-b005-ad7ffae8943c","Type":"ContainerDied","Data":"7b062511dc478830d5a7a95b6a1992b74b4b5e98a580da2e78876fba5dd1b14f"}
Oct 31 04:32:35.608409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.608341    6931 scope.go:122] "RemoveContainer" containerID="bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"
Oct 31 04:32:35.620511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.620492    6931 scope.go:122] "RemoveContainer" containerID="bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"
Oct 31 04:32:35.620848 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:35.620825    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e\": container with ID starting with bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e not found: ID does not exist" containerID="bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"
Oct 31 04:32:35.620918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.620856    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e"} err="failed to get container status \"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e\": rpc error: code = NotFound desc = could not find container \"bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e\": container with ID starting with bb574f7589813f9fca06db1510d08a3321647ef9f92e1123a5a04539f5f2497e not found: ID does not exist"
Oct 31 04:32:35.637767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.637732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:35.662026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.661991    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:32:35.665381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:35.665357    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-81c810-7767ff98c6-fc476"]
Oct 31 04:32:36.122823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:36.122786    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" path="/var/lib/kubelet/pods/5ec7a65f-5ab6-4e44-b005-ad7ffae8943c/volumes"
Oct 31 04:32:36.314179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:36.314048    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:36.526026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:36.525940    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:37.094855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:37.094813    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:37.542506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:37.537637    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:37.762802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:37.762760    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:38.048415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:38.048377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:39.367658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.367621    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:32:39.368133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.367825    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" podUID="ee892a00-0f88-40dd-b431-9d441000a58c" containerName="maas-api" containerID="cri-o://627679d705eccc6d1cef226eec7ceff3ec30b3a071b971d8ddd8e6cb3bbdcc6c" gracePeriod=30
Oct 31 04:32:39.532771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.532678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:39.723380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.723029    6931 generic.go:356] "Generic (PLEG): container finished" podID="ee892a00-0f88-40dd-b431-9d441000a58c" containerID="627679d705eccc6d1cef226eec7ceff3ec30b3a071b971d8ddd8e6cb3bbdcc6c" exitCode=0
Oct 31 04:32:39.723380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.723116    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" event={"ID":"ee892a00-0f88-40dd-b431-9d441000a58c","Type":"ContainerDied","Data":"627679d705eccc6d1cef226eec7ceff3ec30b3a071b971d8ddd8e6cb3bbdcc6c"}
Oct 31 04:32:39.789742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.789713    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:32:39.945889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.945857    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\") pod \"ee892a00-0f88-40dd-b431-9d441000a58c\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") "
Oct 31 04:32:39.946063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.945959    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") pod \"ee892a00-0f88-40dd-b431-9d441000a58c\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") "
Oct 31 04:32:39.946063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.946007    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\") pod \"ee892a00-0f88-40dd-b431-9d441000a58c\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") "
Oct 31 04:32:39.946063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.946046    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\" (UniqueName: \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\") pod \"ee892a00-0f88-40dd-b431-9d441000a58c\" (UID: \"ee892a00-0f88-40dd-b431-9d441000a58c\") "
Oct 31 04:32:39.946456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.946433    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca" pod "ee892a00-0f88-40dd-b431-9d441000a58c" (UID: "ee892a00-0f88-40dd-b431-9d441000a58c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:32:39.946719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.946693    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee892a00-0f88-40dd-b431-9d441000a58c-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:39.948071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.948046    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls" pod "ee892a00-0f88-40dd-b431-9d441000a58c" (UID: "ee892a00-0f88-40dd-b431-9d441000a58c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:39.948282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.948260    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj" pod "ee892a00-0f88-40dd-b431-9d441000a58c" (UID: "ee892a00-0f88-40dd-b431-9d441000a58c"). InnerVolumeSpecName "kube-api-access-62jcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:32:39.948546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:39.948258    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs" pod "ee892a00-0f88-40dd-b431-9d441000a58c" (UID: "ee892a00-0f88-40dd-b431-9d441000a58c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:32:40.048240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.048206    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:40.048240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.048238    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee892a00-0f88-40dd-b431-9d441000a58c-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:40.048240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.048251    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62jcj\" (UniqueName: \"kubernetes.io/projected/ee892a00-0f88-40dd-b431-9d441000a58c-kube-api-access-62jcj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:32:40.322606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.322523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:40.730912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.730835    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"
Oct 31 04:32:40.730912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.730857    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk" event={"ID":"ee892a00-0f88-40dd-b431-9d441000a58c","Type":"ContainerDied","Data":"9523cd471ebdc935bf988d636c1445b96b1fe36d38f195d1598b8d34499a3d9e"}
Oct 31 04:32:40.731481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.730918    6931 scope.go:122] "RemoveContainer" containerID="627679d705eccc6d1cef226eec7ceff3ec30b3a071b971d8ddd8e6cb3bbdcc6c"
Oct 31 04:32:40.759353 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:40.759314    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:32:40.782018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.781987    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:32:40.793990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:40.793958    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e6a0de-9d7d8cd7b-7dqkk"]
Oct 31 04:32:42.125280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:42.125245    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee892a00-0f88-40dd-b431-9d441000a58c" path="/var/lib/kubelet/pods/ee892a00-0f88-40dd-b431-9d441000a58c/volumes"
Oct 31 04:32:42.137830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:42.137661    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:43.770579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.770543    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 04:32:43.771124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771111    6931 cpu_manager.go:414] "containerMap: removing container" podUID="ee892a00-0f88-40dd-b431-9d441000a58c" containerName="maas-api"
Oct 31 04:32:43.771179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771126    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee892a00-0f88-40dd-b431-9d441000a58c" containerName="maas-api"
Oct 31 04:32:43.771213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771187    6931 cpu_manager.go:414] "containerMap: removing container" podUID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" containerName="maas-api"
Oct 31 04:32:43.771213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771193    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" containerName="maas-api"
Oct 31 04:32:43.771213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771203    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy"
Oct 31 04:32:43.771213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771208    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy"
Oct 31 04:32:43.771338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771290    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec7a65f-5ab6-4e44-b005-ad7ffae8943c" containerName="maas-api"
Oct 31 04:32:43.771338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771304    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3ad619-d9db-4a40-8a33-04f861625790" containerName="istio-proxy"
Oct 31 04:32:43.771338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.771317    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee892a00-0f88-40dd-b431-9d441000a58c" containerName="maas-api"
Oct 31 04:32:43.776265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.776245    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:43.797672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.797628    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 04:32:43.811235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.811198    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-57dc6d-serving-cert\""
Oct 31 04:32:43.916992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.902035    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhfj\" (UniqueName: \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:43.916992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.916463    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:43.916992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.916537    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:43.916992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.916610    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:43.970915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:43.970881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:44.017704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.017668    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.017891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.017819    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhfj\" (UniqueName: \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.017891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.017851    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.017891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.017876    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.018067 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:44.018002    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-serving-cert: secret "maas-api-e2e-shared-a-w4-57dc6d-serving-cert" not found
Oct 31 04:32:44.018124 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:32:44.018080    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls podName:38b43381-e151-4f9e-90d7-be115c805ff7 nodeName:}" failed. No retries permitted until 2026-10-31 04:32:44.518062376 +0000 UTC m=+1660.551062033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls") pod "maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" (UID: "38b43381-e151-4f9e-90d7-be115c805ff7") : secret "maas-api-e2e-shared-a-w4-57dc6d-serving-cert" not found
Oct 31 04:32:44.018573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.018548    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.020223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.020203    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.046366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.046335    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhfj\" (UniqueName: \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.214242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.214141    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:44.426543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.426449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:44.526025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.525984    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.528475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.528451    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.690705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.690624    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:44.992249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:44.989933    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:45.041798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.041752    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 04:32:45.047316 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:32:45.047280    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b43381_e151_4f9e_90d7_be115c805ff7.slice/crio-56283308acc583a28ed00959e08774221dcb2a14ff7be15f4685878628fff396 WatchSource:0}: Error finding container 56283308acc583a28ed00959e08774221dcb2a14ff7be15f4685878628fff396: Status 404 returned error can't find the container with id 56283308acc583a28ed00959e08774221dcb2a14ff7be15f4685878628fff396
Oct 31 04:32:45.586182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.581360    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:45.782118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.782070    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" event={"ID":"38b43381-e151-4f9e-90d7-be115c805ff7","Type":"ContainerStarted","Data":"afed65625fc2611d8cb3bdefac5116c019e641b3d710b3b9f1a9665843b12266"}
Oct 31 04:32:45.782315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.782141    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" event={"ID":"38b43381-e151-4f9e-90d7-be115c805ff7","Type":"ContainerStarted","Data":"56283308acc583a28ed00959e08774221dcb2a14ff7be15f4685878628fff396"}
Oct 31 04:32:45.782315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.782192    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:45.825078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:45.825026    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" podStartSLOduration=2.4834158889999998 podStartE2EDuration="2.825010869s" totalImagesPullingTime="341.59498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:32:45.822812003 +0000 UTC m=+1661.855811682" watchObservedRunningTime="2026-10-31 04:32:45.825010869 +0000 UTC m=+1661.858010558"
Oct 31 04:32:46.531104 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:46.531070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:48.524408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:48.524355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:50.435247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:50.435131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:51.804143 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:51.804110    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 04:32:52.454922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:52.454887    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:54.275198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:54.275125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:54.484439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:54.484406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:54.721086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:54.721047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:55.189599 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:55.189550    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:55.405247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:55.405209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:56.214414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:56.214380    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:58.034794 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:58.034756    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:58.418828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:58.418722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:58.639595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:58.639551    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:59.535275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:59.535233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:59.764718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:32:59.764673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:00.072173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:00.072117    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:00.529765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:00.529746    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:00.746126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:00.746091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:01.033234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:01.013874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:01.238114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:01.238080    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:01.474808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:01.474722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:02.942550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:02.942514    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:03.261945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:03.261822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:04.774601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:04.774555    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:05.023621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:05.023524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:05.269538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:05.269504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:05.836756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:05.836712    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:06.047235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:06.047193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:07.818182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:07.818017    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:08.334253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:08.334134    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:08.686441 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:08.686265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:08.922886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:08.922849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:09.367965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:09.367918    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:10.000832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:10.000791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:10.206616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:10.206582    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:10.760244 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:33:10.760209    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:33:11.795207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:11.795140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:13.566606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:13.566569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:13.776456 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:13.776395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:15.294881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:15.294843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:15.536134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:15.536094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:15.801480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:15.801443    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:16.170240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:16.170099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:16.378783 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:16.378744    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:17.595607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.595571    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 04:33:17.601961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.601936    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.619380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.619343    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 04:33:17.686266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.686231    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-031745-serving-cert\""
Oct 31 04:33:17.697583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.697546    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvcl\" (UniqueName: \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.697766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.697727    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.697919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.697902    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.697979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.697964    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.800590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.795829    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:17.808248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.805671    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.808248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.805793    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.808248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.805847    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvcl\" (UniqueName: \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.815189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.811367    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.815189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.812689    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.831364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.831321    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.833448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.833421    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.837960 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.836761    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvcl\" (UniqueName: \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\") pod \"maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:17.915829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:17.915734    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:18.021187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:18.016145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:18.219704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:18.219629    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 04:33:18.223279 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:33:18.223255    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784baa2a_ac6d_4b86_8f03_d85bb7ca50b9.slice/crio-ab781f69ee6eff45c51107d2593a40905a376e208f891bcf4daadd9bb4e375ac WatchSource:0}: Error finding container ab781f69ee6eff45c51107d2593a40905a376e208f891bcf4daadd9bb4e375ac: Status 404 returned error can't find the container with id ab781f69ee6eff45c51107d2593a40905a376e208f891bcf4daadd9bb4e375ac
Oct 31 04:33:18.224664 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:18.224646    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:33:19.185575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:19.185539    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" event={"ID":"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9","Type":"ContainerStarted","Data":"679c940849ca2c4182ff726a11339ed53f6cde1ab153a911aed339eda59ed16a"}
Oct 31 04:33:19.185991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:19.185602    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:19.185991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:19.185620    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" event={"ID":"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9","Type":"ContainerStarted","Data":"ab781f69ee6eff45c51107d2593a40905a376e208f891bcf4daadd9bb4e375ac"}
Oct 31 04:33:19.226883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:19.226797    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" podStartSLOduration=1.866074993 podStartE2EDuration="2.226780784s" totalImagesPullingTime="360.705791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:19.226649837 +0000 UTC m=+1695.259649516" watchObservedRunningTime="2026-10-31 04:33:19.226780784 +0000 UTC m=+1695.259780463"
Oct 31 04:33:20.017786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:20.017750    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:20.243166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:20.243113    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:21.878811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:21.878779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:22.156238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:22.151789    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:22.364784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:22.364746    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:24.072995 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.072953    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 04:33:24.076900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.076875    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.110204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.107053    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-77a153-serving-cert\""
Oct 31 04:33:24.110204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.109780    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 04:33:24.144414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.144368    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.144623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.144538    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgx2\" (UniqueName: \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.144708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.144648    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.144708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.144669    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.246079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.246038    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.246309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.246085    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.246309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.246220    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.246449 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.246353    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgx2\" (UniqueName: \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.246834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.246804    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.248618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.248589    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.248703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.248634    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.310213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.300985    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgx2\" (UniqueName: \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\") pod \"maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.324916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.323770    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:24.391505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.391422    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:24.531897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.531781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:24.751609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:24.751575    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 04:33:24.754130 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:33:24.754105    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f53ce4e_cd08_4d45_9f68_6cf6b027e4e5.slice/crio-19a8441f132aa416180b3a01b066acadfe65a9f7053bb78c2a6e5b7e26246b9d WatchSource:0}: Error finding container 19a8441f132aa416180b3a01b066acadfe65a9f7053bb78c2a6e5b7e26246b9d: Status 404 returned error can't find the container with id 19a8441f132aa416180b3a01b066acadfe65a9f7053bb78c2a6e5b7e26246b9d
Oct 31 04:33:25.221042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:25.221015    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 04:33:25.254888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:25.254846    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" event={"ID":"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5","Type":"ContainerStarted","Data":"259881fc2a1c1e3e51e3820b86b29110816095b6c73196b3e22bc3c050196960"}
Oct 31 04:33:25.255047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:25.254926    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" event={"ID":"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5","Type":"ContainerStarted","Data":"19a8441f132aa416180b3a01b066acadfe65a9f7053bb78c2a6e5b7e26246b9d"}
Oct 31 04:33:25.255232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:25.255213    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:25.309590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:25.309529    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" podStartSLOduration=0.907228344 podStartE2EDuration="1.309510556s" totalImagesPullingTime="402.282212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:25.305847144 +0000 UTC m=+1701.338846826" watchObservedRunningTime="2026-10-31 04:33:25.309510556 +0000 UTC m=+1701.342510236"
Oct 31 04:33:26.305530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:26.305491    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:27.873600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.873568    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 04:33:27.877886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.877864    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:27.900931 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.900901    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 04:33:27.908652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.908605    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-b2a372-serving-cert\""
Oct 31 04:33:27.909913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.909888    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qlx\" (UniqueName: \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:27.910073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.910014    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:27.910182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.910125    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:27.910267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:27.910250    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.012034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.011998    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.012286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.012072    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.012286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.012147    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qlx\" (UniqueName: \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.012286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.012240    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.012469 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:33:28.012424    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-serving-cert: secret "maas-api-e2e-shared-b-w3-b2a372-serving-cert" not found
Oct 31 04:33:28.012527 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:33:28.012500    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls podName:74ccaa85-2ff7-4212-a176-bf6645a0eeae nodeName:}" failed. No retries permitted until 2026-10-31 04:33:28.512479866 +0000 UTC m=+1704.545479524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls") pod "maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" (UID: "74ccaa85-2ff7-4212-a176-bf6645a0eeae") : secret "maas-api-e2e-shared-b-w3-b2a372-serving-cert" not found
Oct 31 04:33:28.012861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.012839    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.014739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.014714    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.035630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.035592    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qlx\" (UniqueName: \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.072109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.072066    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:28.308179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.301568    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:28.548074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.548031    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.550520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.550494    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:28.793846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:28.793814    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:29.182578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:29.182489    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 04:33:29.318419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:29.318377    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" event={"ID":"74ccaa85-2ff7-4212-a176-bf6645a0eeae","Type":"ContainerStarted","Data":"d2c5efdfed3250c849a8ae9d3464ae20a5973c7838d6177f9951ac65989fa686"}
Oct 31 04:33:30.010964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:30.010875    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:30.346178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:30.344067    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" event={"ID":"74ccaa85-2ff7-4212-a176-bf6645a0eeae","Type":"ContainerStarted","Data":"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"}
Oct 31 04:33:30.346178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:30.345202    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:30.396181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:30.395373    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" podStartSLOduration=3.038050171 podStartE2EDuration="3.39535392s" totalImagesPullingTime="357.303749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:30.392015746 +0000 UTC m=+1706.425015426" watchObservedRunningTime="2026-10-31 04:33:30.39535392 +0000 UTC m=+1706.428353601"
Oct 31 04:33:31.292361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:31.292330    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 04:33:32.019663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:32.019625    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:33.882526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:33.882490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:35.900050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:35.900014    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:37.369676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:37.369644    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 04:33:37.879310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:37.879197    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:38.096297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:38.096261    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:38.322911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:38.322877    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:39.897580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:39.897517    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:40.251455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.251371    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 04:33:40.256194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.256139    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.279579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.279544    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 04:33:40.350490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350459    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350514    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350622    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8fv\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350686    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350754    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350776    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350811    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.350974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350957    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.351009 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.350993    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.362422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.362392    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-031745-openshift-default-dockercfg-q5sr5\""
Oct 31 04:33:40.452432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452400    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452445    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452499    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452541    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452560    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8fv\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452581    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452616    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452644    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452697    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.452897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452814    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.453092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.452996    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.453092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.453012    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.453092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.453067    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.453468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.453444    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.455112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.455092    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.455262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.455246    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.479219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.479185    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.479338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.479293    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8fv\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.578647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.578611    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:40.760320 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:33:40.760282    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:33:40.912938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.912859    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 04:33:40.916347 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:33:40.916321    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95d4201_a147_411b_b427_db5a2f1f7a9e.slice/crio-1377d7b391bd50cfc86a5792aaafac03d1caafc7360ba050c88929e444e0caf0 WatchSource:0}: Error finding container 1377d7b391bd50cfc86a5792aaafac03d1caafc7360ba050c88929e444e0caf0: Status 404 returned error can't find the container with id 1377d7b391bd50cfc86a5792aaafac03d1caafc7360ba050c88929e444e0caf0
Oct 31 04:33:40.918949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.918917    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:40.919036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.918985    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:40.919036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:40.919016    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:41.428915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.428874    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" event={"ID":"f95d4201-a147-411b-b427-db5a2f1f7a9e","Type":"ContainerStarted","Data":"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"}
Oct 31 04:33:41.429115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.428940    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" event={"ID":"f95d4201-a147-411b-b427-db5a2f1f7a9e","Type":"ContainerStarted","Data":"1377d7b391bd50cfc86a5792aaafac03d1caafc7360ba050c88929e444e0caf0"}
Oct 31 04:33:41.475289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.475228    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podStartSLOduration=1.475211883 podStartE2EDuration="1.475211883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:41.475031775 +0000 UTC m=+1717.508031474" watchObservedRunningTime="2026-10-31 04:33:41.475211883 +0000 UTC m=+1717.508211562"
Oct 31 04:33:41.579396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.579364    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:41.581075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.581047    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:41.581210 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.581101    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:41.805078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:41.805047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:42.579206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:42.579144    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:42.579662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:42.579239    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:43.579349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:43.579313    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:43.579743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:43.579365    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:43.754201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:43.754105    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:44.579737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:44.579663    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:44.579737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:44.579722    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:44.720186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:44.720115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:45.570315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.570281    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:33:45.574102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.574082    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.578874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.578851    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:45.578996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.578891    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:45.602758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.602719    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7229f2-serving-cert\""
Oct 31 04:33:45.605643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.605616    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:33:45.656972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.656931    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.657588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.657563    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg69\" (UniqueName: \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.657731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.657714    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.657802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.657773    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.758774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.758730    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.758988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.758790    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.758988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.758912    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg69\" (UniqueName: \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.759115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.758994    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.759634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.759612    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.761429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.761405    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.761521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.761412    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.779007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.778980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:45.781918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.781898    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg69\" (UniqueName: \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\") pod \"maas-api-e2e-autores-7229f2-68fbf7878-ljrrp\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:45.891236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:45.889648    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:46.001253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.001095    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:46.096331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.096297    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:33:46.102944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.102915    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.189082 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.189012    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:33:46.268217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268182    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268223    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2x8\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268392    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268450    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268616    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268637    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268743    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268795    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.268845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.268829    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.281258 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:33:46.281217    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0af6f7_86af_4cec_8008_a1f722f9fe74.slice/crio-30ccc53fe4233cfac2c6976fe13afd7c30b02b783812c07e661854e566c06c4d WatchSource:0}: Error finding container 30ccc53fe4233cfac2c6976fe13afd7c30b02b783812c07e661854e566c06c4d: Status 404 returned error can't find the container with id 30ccc53fe4233cfac2c6976fe13afd7c30b02b783812c07e661854e566c06c4d
Oct 31 04:33:46.281866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.281846    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:33:46.370558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370520    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.370558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370563    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.370801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370586    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.370801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370652    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.370801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370670    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2x8\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.370801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370792    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370841    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370918    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.370945    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.371048    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.371204    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.371307    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.371352    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.371603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.371470    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.372828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.372809    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.373278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.373263    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.393414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.393388    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.393562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.393544    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2x8\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\") pod \"e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") " pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.417959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.417925    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:46.484804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.484767    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" event={"ID":"7b0af6f7-86af-4cec-8008-a1f722f9fe74","Type":"ContainerStarted","Data":"30ccc53fe4233cfac2c6976fe13afd7c30b02b783812c07e661854e566c06c4d"}
Oct 31 04:33:46.579452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.579417    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 04:33:46.579646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.579490    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 04:33:46.801221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.801115    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:33:46.817900 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:33:46.817535    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbac4ab_c725_48cb_a326_d17f6aeaa09b.slice/crio-92f0186cce2709551f648b39359548e9a88b619cd8aeac63eb4b44a64536d7fc WatchSource:0}: Error finding container 92f0186cce2709551f648b39359548e9a88b619cd8aeac63eb4b44a64536d7fc: Status 404 returned error can't find the container with id 92f0186cce2709551f648b39359548e9a88b619cd8aeac63eb4b44a64536d7fc
Oct 31 04:33:46.838407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.828509    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:46.838407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.828605    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:46.838407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:46.828650    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:33:47.504803 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.504761    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" event={"ID":"7b0af6f7-86af-4cec-8008-a1f722f9fe74","Type":"ContainerStarted","Data":"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"}
Oct 31 04:33:47.505006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.504887    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:47.507541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.507200    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" event={"ID":"6dbac4ab-c725-48cb-a326-d17f6aeaa09b","Type":"ContainerStarted","Data":"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"}
Oct 31 04:33:47.507541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.507276    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" event={"ID":"6dbac4ab-c725-48cb-a326-d17f6aeaa09b","Type":"ContainerStarted","Data":"92f0186cce2709551f648b39359548e9a88b619cd8aeac63eb4b44a64536d7fc"}
Oct 31 04:33:47.549951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.549874    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" podStartSLOduration=2.250896658 podStartE2EDuration="2.549854601s" totalImagesPullingTime="298.957943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:47.546205992 +0000 UTC m=+1723.579205689" watchObservedRunningTime="2026-10-31 04:33:47.549854601 +0000 UTC m=+1723.582854281"
Oct 31 04:33:47.675864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.675536    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:47.691184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.677053    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:47.721891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.720955    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 04:33:47.751190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:47.750928    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podStartSLOduration=1.750908419 podStartE2EDuration="1.750908419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:47.596335539 +0000 UTC m=+1723.629335215" watchObservedRunningTime="2026-10-31 04:33:47.750908419 +0000 UTC m=+1723.783908101"
Oct 31 04:33:48.175101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:48.175061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:48.419080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:48.419042    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:48.420792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:48.420767    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:48.420918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:48.420833    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:49.418645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:49.418601    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:49.419070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:49.418678    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:50.082884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:50.082270    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:50.333935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:50.333836    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:50.418292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:50.418261    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:50.418480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:50.418311    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:51.419248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:51.419209    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:51.419847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:51.419273    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:52.061886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:52.061847    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:52.418697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:52.418616    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:52.418697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:52.418674    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:52.455180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:52.451605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:53.419125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:53.419077    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 04:33:53.419324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:53.419173    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 04:33:53.520540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:53.520508    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:33:54.149828 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:54.149786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:54.580470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:54.580437    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:54.580959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:54.580893    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:54.581700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:54.581679    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:33:56.223276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:56.223234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:58.188364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:58.188326    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:58.412779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:58.412739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:58.619936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:58.619879    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:58.843426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:58.843363    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:59.321417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:59.321373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:59.527646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:33:59.527605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:00.442659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:00.442621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:01.552413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:01.552370    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:02.851781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:02.851742    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:03.595127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:03.595092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:03.815462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:03.815418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:05.032452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:05.032410    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:05.848380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:05.848343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:06.092193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:06.092136    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:07.262568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:07.262528    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:09.324923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:09.324890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:10.760181 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:34:10.760116    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:34:11.246547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:11.246513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:13.103801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:13.103761    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:14.791477 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:14.791438    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:15.071833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:15.071741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:17.134889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:17.134852    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:19.096481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:19.096442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:19.320684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:19.320651    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:20.942822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:20.942789    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:21.158589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:21.158556    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:22.932693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:22.932655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:24.820398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:24.820338    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:25.377729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.377695    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:34:25.401179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.391492    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.426143 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.425594    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:34:25.485247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485216    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485259    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485279    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485396    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485441    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485560    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485666    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485774    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.485870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.485856    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58p8\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588655    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58p8\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588762    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588800    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588830    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588885    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588933    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.588998    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.589082    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.589194    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.590188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.589760    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.594184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.593929    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.594775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.594466    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.597107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.597084    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.597246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.597140    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.597349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.597326    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.597539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.597517    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.613724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.613687    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58p8\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.619501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.619468    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\") pod \"e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") " pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:25.778614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:25.778523    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:26.102877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.102842    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:34:26.106305 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:34:26.106272    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717b6bab_513f_46e5_b202_30c8c71bf5cf.slice/crio-660cad8e02a6003d5b928bd1a015beb0c7139778beb133cae7f4404db878f5d6 WatchSource:0}: Error finding container 660cad8e02a6003d5b928bd1a015beb0c7139778beb133cae7f4404db878f5d6: Status 404 returned error can't find the container with id 660cad8e02a6003d5b928bd1a015beb0c7139778beb133cae7f4404db878f5d6
Oct 31 04:34:26.108892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.108851    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:34:26.108961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.108940    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:34:26.108998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.108978    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:34:26.704706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.704663    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:26.839411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.839379    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" event={"ID":"717b6bab-513f-46e5-b202-30c8c71bf5cf","Type":"ContainerStarted","Data":"b67bcd74b0095c0cf7e176e6fb66769fd5070f72840f3f80d78d6635aa57124b"}
Oct 31 04:34:26.839595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.839457    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" event={"ID":"717b6bab-513f-46e5-b202-30c8c71bf5cf","Type":"ContainerStarted","Data":"660cad8e02a6003d5b928bd1a015beb0c7139778beb133cae7f4404db878f5d6"}
Oct 31 04:34:26.907453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:26.907379    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podStartSLOduration=1.907363583 podStartE2EDuration="1.907363583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:26.894937303 +0000 UTC m=+1762.927936999" watchObservedRunningTime="2026-10-31 04:34:26.907363583 +0000 UTC m=+1762.940363262"
Oct 31 04:34:27.779136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:27.779098    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:27.780467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:27.780442    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 04:34:27.780593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:27.780489    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 04:34:28.486529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:28.486487    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:28.779269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:28.779180    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 04:34:28.779269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:28.779235    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 04:34:29.779110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:29.779078    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 04:34:29.779339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:29.779135    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 04:34:29.957928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:29.957890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:30.395072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.395035    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 04:34:30.399596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.399569    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.437083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.436992    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-d6ac82-serving-cert\""
Oct 31 04:34:30.443281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.443230    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 04:34:30.481713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.481671    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.482034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.482015    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.482303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.482282    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgnp\" (UniqueName: \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.482482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.482455    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.583886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.583840    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgnp\" (UniqueName: \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.584268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.584233    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.584787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.584765    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.585171 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:34:30.585000    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-serving-cert: secret "maas-api-e2e-shared-b-w1-d6ac82-serving-cert" not found
Oct 31 04:34:30.585710 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:34:30.585689    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls podName:6af0032f-6e1f-44c3-946a-cf96ef798034 nodeName:}" failed. No retries permitted until 2026-10-31 04:34:31.085658092 +0000 UTC m=+1767.118657755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls") pod "maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" (UID: "6af0032f-6e1f-44c3-946a-cf96ef798034") : secret "maas-api-e2e-shared-b-w1-d6ac82-serving-cert" not found
Oct 31 04:34:30.585949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.585928    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.586857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.586736    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.618183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.611602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:30.618636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.618555    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.638364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.637703    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgnp\" (UniqueName: \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:30.779800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.779774    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 04:34:30.780236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.779829    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 04:34:30.851887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:30.851848    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:31.059661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.059572    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:31.094100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.094059    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:31.096619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.096596    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:31.315221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.315111    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:31.669714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.669629    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 04:34:31.779453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.779419    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 04:34:31.779618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.779516    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 04:34:31.934474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:31.934297    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" event={"ID":"6af0032f-6e1f-44c3-946a-cf96ef798034","Type":"ContainerStarted","Data":"3fca809168bd6b4db6be7c3f8a353ea319fbfb142d45e7914bcddc4d2b915f49"}
Oct 31 04:34:32.794842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.794796    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:32.806196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.804582    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:32.806196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.805831    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:32.808744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.808716    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:34:32.946901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.946858    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" event={"ID":"6af0032f-6e1f-44c3-946a-cf96ef798034","Type":"ContainerStarted","Data":"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"}
Oct 31 04:34:32.947478 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.947113    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:32.990594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:32.990537    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" podStartSLOduration=2.603788985 podStartE2EDuration="2.990521004s" totalImagesPullingTime="386.732019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:32.989735031 +0000 UTC m=+1769.022734710" watchObservedRunningTime="2026-10-31 04:34:32.990521004 +0000 UTC m=+1769.023520682"
Oct 31 04:34:34.894768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:34.894729    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:36.770256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:36.770215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:38.494075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:38.494033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:38.988649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:38.988621    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 04:34:40.267018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:40.266979    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:40.761789 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:34:40.761754    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:34:42.058946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:42.058889    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:43.462330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:43.462284    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:44.031932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:44.031894    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:45.913109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:45.913067    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:47.812646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:47.812613    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:48.824425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:48.824384    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:49.732872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:49.732830    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:51.502669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:51.502634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:53.314117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:53.314064    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:55.077645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:55.077603    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:55.493935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:55.493831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:56.932614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:56.932551    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:58.658497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:58.658461    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:58.867485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:34:58.867410    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:00.611144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:00.611107    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:02.724181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:02.721821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:04.698435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:04.698388    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:04.908491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:04.908451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.533227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:06.533131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.740897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:06.740862    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:08.029618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:08.029565    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"]
Oct 31 04:35:08.041466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:08.041429    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c162de-jmh69"]
Oct 31 04:35:08.123074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:08.123039    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39212ede-3fbc-4c7c-a83b-23c9334064e1" path="/var/lib/kubelet/pods/39212ede-3fbc-4c7c-a83b-23c9334064e1/volumes"
Oct 31 04:35:08.459822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:08.459781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:10.229323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:10.229285    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:10.763240 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:35:10.763204    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:35:12.002947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:12.002907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:13.786323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:13.786286    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:15.523774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:15.523735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:15.728026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:15.727993    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:17.464077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:17.464031    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:19.325107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:19.325067    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:21.127046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:21.127010    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:22.918034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:22.917995    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:23.260523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:23.260419    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:24.803657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:24.803611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:26.715634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:26.715593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:28.503680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:28.503636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:30.256060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:30.256019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:32.050691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:32.050651    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:33.801909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:33.801868    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:35.547350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:35.547256    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:36.425902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:36.425852    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:36.850647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:36.850608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:37.493684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:37.493646    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:39.503409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:39.503368    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:40.760199 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:35:40.760138    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:35:40.775545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:40.775504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:40.997864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:40.997824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:41.532494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:41.532453    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:41.879690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:41.879595    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:43.423458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:43.423417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:45.210997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:45.210957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:46.994298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:46.994130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:48.800435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:48.800399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:50.497573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:50.497521    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:52.350677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:52.350644    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:54.070946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:54.070902    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:55.914096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:55.914059    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:57.777409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:57.777374    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:58.771622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:58.771583    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:59.594577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:35:59.594531    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:01.512134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:01.512068    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:03.092964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:03.092921    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:03.438238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:03.437459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:04.703960 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:04.703931    6931 scope.go:122] "RemoveContainer" containerID="ec8c71a5134ee4f7e9fcebca14b8118dadb47739d8afcbf04f6b503bb9a34327"
Oct 31 04:36:05.642791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:05.642753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:07.577525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:07.577489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:08.160463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:08.160413    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:08.367781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:08.367744    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:09.586846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:09.586800    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:10.761453 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:36:10.761416    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:36:11.634146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:11.634113    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:13.513340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:13.513300    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:15.408432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:15.408394    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:17.291215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:17.291173    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:18.243238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:18.243199    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:18.453092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:18.453058    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:19.289139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:19.289100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:19.500329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:19.500294    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:21.307410 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:21.307371    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:23.099436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:23.099385    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:24.904230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:24.904140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:25.320124 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:25.320090    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:26.670716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:26.670674    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:28.493374 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:28.493338    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:30.305772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:30.305626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:32.068791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:32.068740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:33.886021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:33.885989    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:35.287737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.287680    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:36:35.288176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.287927    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" podUID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" containerName="maas-api" containerID="cri-o://c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e" gracePeriod=30
Oct 31 04:36:35.458308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.458267    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:35.723626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.723543    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:36:35.906597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.904178    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\") pod \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") "
Oct 31 04:36:35.906947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.906920    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\" (UniqueName: \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\") pod \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") "
Oct 31 04:36:35.909190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.907682    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca" pod "7b0af6f7-86af-4cec-8008-a1f722f9fe74" (UID: "7b0af6f7-86af-4cec-8008-a1f722f9fe74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:36:35.909190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.907150    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\") pod \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") "
Oct 31 04:36:35.909190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.907882    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\") pod \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\" (UID: \"7b0af6f7-86af-4cec-8008-a1f722f9fe74\") "
Oct 31 04:36:35.909190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.908568    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0af6f7-86af-4cec-8008-a1f722f9fe74-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:36:35.910096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.910064    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69" pod "7b0af6f7-86af-4cec-8008-a1f722f9fe74" (UID: "7b0af6f7-86af-4cec-8008-a1f722f9fe74"). InnerVolumeSpecName "kube-api-access-jvg69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:35.915351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.915315    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs" pod "7b0af6f7-86af-4cec-8008-a1f722f9fe74" (UID: "7b0af6f7-86af-4cec-8008-a1f722f9fe74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:36:35.916408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.916371    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls" pod "7b0af6f7-86af-4cec-8008-a1f722f9fe74" (UID: "7b0af6f7-86af-4cec-8008-a1f722f9fe74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:36:35.959715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.959679    6931 generic.go:356] "Generic (PLEG): container finished" podID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" containerID="c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e" exitCode=0
Oct 31 04:36:35.959919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.959731    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"
Oct 31 04:36:35.959919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.959750    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" event={"ID":"7b0af6f7-86af-4cec-8008-a1f722f9fe74","Type":"ContainerDied","Data":"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"}
Oct 31 04:36:35.959919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.959795    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp" event={"ID":"7b0af6f7-86af-4cec-8008-a1f722f9fe74","Type":"ContainerDied","Data":"30ccc53fe4233cfac2c6976fe13afd7c30b02b783812c07e661854e566c06c4d"}
Oct 31 04:36:35.959919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.959815    6931 scope.go:122] "RemoveContainer" containerID="c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"
Oct 31 04:36:35.973827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.973799    6931 scope.go:122] "RemoveContainer" containerID="c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"
Oct 31 04:36:35.974196 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:36:35.974150    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e\": container with ID starting with c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e not found: ID does not exist" containerID="c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"
Oct 31 04:36:35.974275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:35.974216    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e"} err="failed to get container status \"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e\": rpc error: code = NotFound desc = could not find container \"c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e\": container with ID starting with c4a40ea1faaae768e79308fc8c1832329e4f5dfc632ba9998bf71612e4fa994e not found: ID does not exist"
Oct 31 04:36:36.010367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.010326    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvg69\" (UniqueName: \"kubernetes.io/projected/7b0af6f7-86af-4cec-8008-a1f722f9fe74-kube-api-access-jvg69\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:36:36.010367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.010368    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:36:36.010367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.010377    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0af6f7-86af-4cec-8008-a1f722f9fe74-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:36:36.011708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.011683    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:36:36.017774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.017748    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7229f2-68fbf7878-ljrrp"]
Oct 31 04:36:36.123184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.123141    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" path="/var/lib/kubelet/pods/7b0af6f7-86af-4cec-8008-a1f722f9fe74/volumes"
Oct 31 04:36:36.415518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:36.415477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:38.210837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:38.210801    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:40.144286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:40.144245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:40.360713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:40.360675    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:40.759743 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:36:40.759706    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:36:40.890790 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:40.890747    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:41.357200 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:41.353848    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:41.990883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:41.990851    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:42.209506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:42.209470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:42.488645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:42.488611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:42.914791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:42.914748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:43.917107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:43.914482    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:44.136561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:44.136481    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:44.365596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:44.365557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:46.032346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.032308    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:46.579894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.579859    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:36:46.580904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.580881    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" containerName="maas-api"
Oct 31 04:36:46.580904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.580905    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" containerName="maas-api"
Oct 31 04:36:46.581323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.581069    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0af6f7-86af-4cec-8008-a1f722f9fe74" containerName="maas-api"
Oct 31 04:36:46.586281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.586260    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.605902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.605867    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:36:46.610877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.610836    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-575fe8\"/\"kube-root-ca.crt\""
Oct 31 04:36:46.613413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.613384    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-575fe8\"/\"noten-model-575fe8-kserve-self-signed-certs\""
Oct 31 04:36:46.613537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.613443    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-575fe8\"/\"openshift-service-ca.crt\""
Oct 31 04:36:46.613537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.613442    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-575fe8\"/\"default-dockercfg-vtlln\""
Oct 31 04:36:46.689583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689536    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.689752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689733    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.689840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689806    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.689981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689904    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.689981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689943    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.689981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.689961    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98bp\" (UniqueName: \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.731944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.731904    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:46.791917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.791882    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.791935    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.791973    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.791989    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98bp\" (UniqueName: \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.792071    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.792223    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.792483    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.792673    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.792749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.792712    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.795122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.795084    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.801857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.795984    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.820017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.819771    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98bp\" (UniqueName: \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\") pod \"noten-model-575fe8-kserve-6ffbc9f65c-jzdpv\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") " pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:46.915899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:46.915437    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:47.301095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:47.301063    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:36:47.304626 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:36:47.304597    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36300b4c_ca07_4238_ab18_1bdcb9d25ee8.slice/crio-2799e4aedf98884ae108a0295f8241be9180e438382d650edd0c20c92817c373 WatchSource:0}: Error finding container 2799e4aedf98884ae108a0295f8241be9180e438382d650edd0c20c92817c373: Status 404 returned error can't find the container with id 2799e4aedf98884ae108a0295f8241be9180e438382d650edd0c20c92817c373
Oct 31 04:36:48.065212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:48.060703    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" event={"ID":"36300b4c-ca07-4238-ab18-1bdcb9d25ee8","Type":"ContainerStarted","Data":"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"}
Oct 31 04:36:48.065212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:48.060763    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" event={"ID":"36300b4c-ca07-4238-ab18-1bdcb9d25ee8","Type":"ContainerStarted","Data":"2799e4aedf98884ae108a0295f8241be9180e438382d650edd0c20c92817c373"}
Oct 31 04:36:48.100146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:48.100096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:48.103544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:48.103490    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" podStartSLOduration=2.103479172 podStartE2EDuration="2.103479172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:36:48.10322349 +0000 UTC m=+1904.136223173" watchObservedRunningTime="2026-10-31 04:36:48.103479172 +0000 UTC m=+1904.136478850"
Oct 31 04:36:48.917184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:48.917119    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:50.013862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:50.013815    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:51.799139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:51.799103    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:53.692631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:53.692589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:55.438502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:55.438457    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:56.916435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:56.916405    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:56.929319 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:56.929288    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:57.128908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:57.128881    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:36:57.246033 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:57.245930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:58.904100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:58.904060    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:59.134369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:36:59.134325    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:00.830818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:00.830773    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:01.833152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:01.833114    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:02.654405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:02.654361    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:04.493774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:04.493723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:06.360733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:06.360695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:08.206862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:08.206822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:10.000860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:10.000824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:10.759139 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:37:10.759091    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:37:11.239127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:11.239086    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:37:11.386469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:11.386432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:11.639265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:11.639229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:11.887769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:11.887728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:12.100643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:12.100602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:12.309095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:12.309061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:13.941878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:13.941839    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:14.656130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:14.656084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:14.860649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:14.860602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:15.688057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:15.688020    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:17.072013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:17.071971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:17.399875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:17.399781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:19.318035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:19.317998    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:21.223515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:21.223480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:23.105207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:23.105127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:24.853794 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:24.853744    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:26.263107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.263047    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" podUID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" containerName="main" containerID="cri-o://e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7" gracePeriod=45
Oct 31 04:37:26.551320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.551295    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:37:26.636903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.636856    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.637084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.636940    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.637084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637036    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.637261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637140    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.637261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637216    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:37:26.637261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637235    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\" (UniqueName: \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.637478 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637430    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:37:26.637557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.637290    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\") pod \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\" (UID: \"36300b4c-ca07-4238-ab18-1bdcb9d25ee8\") "
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.638049    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.638077    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.638267    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.639873    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.640144    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "kube-api-access-l98bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:37:26.645099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.640481    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs" pod "36300b4c-ca07-4238-ab18-1bdcb9d25ee8" (UID: "36300b4c-ca07-4238-ab18-1bdcb9d25ee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:37:26.742206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.739896    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l98bp\" (UniqueName: \"kubernetes.io/projected/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-kube-api-access-l98bp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.742206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.739937    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.742206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.739955    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.742206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.739967    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36300b4c-ca07-4238-ab18-1bdcb9d25ee8-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:37:26.758078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:26.758045    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:27.358813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.358778    6931 generic.go:356] "Generic (PLEG): container finished" podID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" containerID="e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7" exitCode=0
Oct 31 04:37:27.359348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.358821    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"
Oct 31 04:37:27.359348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.358868    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" event={"ID":"36300b4c-ca07-4238-ab18-1bdcb9d25ee8","Type":"ContainerDied","Data":"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"}
Oct 31 04:37:27.359348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.358917    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv" event={"ID":"36300b4c-ca07-4238-ab18-1bdcb9d25ee8","Type":"ContainerDied","Data":"2799e4aedf98884ae108a0295f8241be9180e438382d650edd0c20c92817c373"}
Oct 31 04:37:27.359348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.358942    6931 scope.go:122] "RemoveContainer" containerID="e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"
Oct 31 04:37:27.369335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.369316    6931 scope.go:122] "RemoveContainer" containerID="e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"
Oct 31 04:37:27.369629 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:37:27.369609    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7\": container with ID starting with e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7 not found: ID does not exist" containerID="e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"
Oct 31 04:37:27.369694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.369642    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7"} err="failed to get container status \"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7\": rpc error: code = NotFound desc = could not find container \"e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7\": container with ID starting with e434435ee257bb35500e2ca125e1b0d3a8b9dd46df8649f9f3ce01959b5620d7 not found: ID does not exist"
Oct 31 04:37:27.411900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.411866    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:37:27.423209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:27.423178    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-575fe8/noten-model-575fe8-kserve-6ffbc9f65c-jzdpv"]
Oct 31 04:37:28.124248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:28.124218    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" path="/var/lib/kubelet/pods/36300b4c-ca07-4238-ab18-1bdcb9d25ee8/volumes"
Oct 31 04:37:28.564216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:28.564178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:30.289352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:30.289315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:32.006351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:32.006302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:33.779874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:33.779835    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:35.623800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:35.623761    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:37.422521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:37.422482    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:39.296554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:39.296510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:40.760345 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:37:40.760309    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:37:41.226181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:41.226128    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:43.114603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:43.114566    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:44.850836 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:44.850798    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:46.599620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:46.599579    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:48.358059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:48.358018    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:48.585938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:48.585896    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:50.144002 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:50.143961    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:51.907551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:51.907515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:53.803327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:53.803275    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:54.385633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:54.385589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:55.613694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:55.613655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:57.496682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:57.496649    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:59.019791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:59.019749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:59.239569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:59.239535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:59.501366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:59.501329    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:59.706659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:59.706619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:59.995186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:37:59.995140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:00.222180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:00.221346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:01.534696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:01.534659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:03.576404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:03.576345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:03.967129 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:03.967047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:04.175913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:04.175877    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:05.484808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:05.484752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:05.759201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:05.759093    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:06.015008 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:06.014900    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:06.917260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:06.917225    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:38:06.917676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:06.917422    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy" containerID="cri-o://3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9" gracePeriod=30
Oct 31 04:38:07.396269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:07.396233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:09.230896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:09.230857    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:10.758898 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:10.758861    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:38:11.102494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:11.102455    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:11.672127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:11.672093    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:38:11.672333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:11.672312    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy" containerID="cri-o://c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b" gracePeriod=30
Oct 31 04:38:12.226831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.226803    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:38:12.339746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339712    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.339925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339768    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.339925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339884    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.339925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339909    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339968    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.339977    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:12.340201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340058    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340081    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340125    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340188    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\") pod \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\" (UID: \"0ef3cd76-b15a-4b81-8c64-e56b33633fa8\") "
Oct 31 04:38:12.340408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340228    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:12.340408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340268    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:12.340522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340423    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:12.340640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340615    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:12.340896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340865    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.340957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340904    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.340957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340920    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.340957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340933    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.340957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.340947    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.342337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.342315    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "kube-api-access-m7h7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:12.342425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.342332    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:12.342609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.342586    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:38:12.342703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.342633    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token" pod "0ef3cd76-b15a-4b81-8c64-e56b33633fa8" (UID: "0ef3cd76-b15a-4b81-8c64-e56b33633fa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:12.442036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.441956    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7h7j\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-kube-api-access-m7h7j\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.442036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.441987    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.442036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.441997    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.442036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.442009    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef3cd76-b15a-4b81-8c64-e56b33633fa8-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:12.743833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.743755    6931 generic.go:356] "Generic (PLEG): container finished" podID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerID="3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9" exitCode=0
Oct 31 04:38:12.743833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.743798    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"
Oct 31 04:38:12.744019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.743830    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" event={"ID":"0ef3cd76-b15a-4b81-8c64-e56b33633fa8","Type":"ContainerDied","Data":"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"}
Oct 31 04:38:12.744019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.743882    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76" event={"ID":"0ef3cd76-b15a-4b81-8c64-e56b33633fa8","Type":"ContainerDied","Data":"b098a29694c0cc4cf86d77c2603ab1acb1907b84b19e022b86ae3f93921c7890"}
Oct 31 04:38:12.744019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.743906    6931 scope.go:122] "RemoveContainer" containerID="3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"
Oct 31 04:38:12.754797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.754776    6931 scope.go:122] "RemoveContainer" containerID="3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"
Oct 31 04:38:12.755075 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:12.755056    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9\": container with ID starting with 3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9 not found: ID does not exist" containerID="3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"
Oct 31 04:38:12.755139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.755085    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9"} err="failed to get container status \"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9\": rpc error: code = NotFound desc = could not find container \"3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9\": container with ID starting with 3e5b23d8828a8824347cd6fef6b8b63789814b99ec9df3e169005bcb15b7f1e9 not found: ID does not exist"
Oct 31 04:38:12.796842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.796811    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:38:12.809799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.809768    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kcm76"]
Oct 31 04:38:12.894848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:12.894805    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:14.124848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:14.124773    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" path="/var/lib/kubelet/pods/0ef3cd76-b15a-4b81-8c64-e56b33633fa8/volumes"
Oct 31 04:38:14.714325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:14.714283    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:16.533209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.533127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:16.864874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.864840    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:38:16.865495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865476    6931 cpu_manager.go:414] "containerMap: removing container" podUID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" containerName="main"
Oct 31 04:38:16.865495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865492    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" containerName="main"
Oct 31 04:38:16.865682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865510    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy"
Oct 31 04:38:16.865682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865515    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy"
Oct 31 04:38:16.865682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865606    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="36300b4c-ca07-4238-ab18-1bdcb9d25ee8" containerName="main"
Oct 31 04:38:16.865682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.865616    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef3cd76-b15a-4b81-8c64-e56b33633fa8" containerName="istio-proxy"
Oct 31 04:38:16.869116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.869097    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:16.911753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.911719    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:38:16.967221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.967186    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:38:16.986878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:16.986841    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b2a372-openshift-default-dockercfg-44djb\""
Oct 31 04:38:17.000447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000416    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000502    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000544    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000698    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000731    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000763    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000781    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000801    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.000983 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.000960    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8k4g\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.102291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102198    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102244    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102298    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102361    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102407    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102475    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102530    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:17.102831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102733    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102758    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:17.102831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102806    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102844    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\") pod \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\" (UID: \"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d\") "
Oct 31 04:38:17.102984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.102846    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:17.103093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103023    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:17.103093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103037    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:17.103093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103071    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103106    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103176    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103203    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103232    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103320    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8k4g\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103473    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103501    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103511    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103533    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.103873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103850    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.103873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103875    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.104056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103889    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.104056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103906    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.104056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103921    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.104056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.103991    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.104400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.104378    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.105216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.104563    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.105216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.105146    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:38:17.105398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.105370    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.105713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.105689    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:17.105810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.105785    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "kube-api-access-j7hn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:17.105879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.105860    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token" pod "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" (UID: "2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:17.106729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.106709    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.106803 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.106748    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.128048 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.128011    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.128205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.128105    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8k4g\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\") pod \"e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") " pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.183352 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.183311    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:17.205021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.204990    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.205196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.205025    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7hn4\" (UniqueName: \"kubernetes.io/projected/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-kube-api-access-j7hn4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.205196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.205041    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.205196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.205053    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:17.524980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.524895    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:38:17.528977 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:38:17.528937    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e0980a_4a4a_49cc_9627_07c8f6f460cb.slice/crio-b7227e807a2b35248578c40148c570a6faaf842d26461c8fca07ff4e7f4b1045 WatchSource:0}: Error finding container b7227e807a2b35248578c40148c570a6faaf842d26461c8fca07ff4e7f4b1045: Status 404 returned error can't find the container with id b7227e807a2b35248578c40148c570a6faaf842d26461c8fca07ff4e7f4b1045
Oct 31 04:38:17.531571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.531534    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:38:17.531676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.531613    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:38:17.531676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.531645    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:38:17.780050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.780014    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" event={"ID":"72e0980a-4a4a-49cc-9627-07c8f6f460cb","Type":"ContainerStarted","Data":"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"}
Oct 31 04:38:17.780601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.780078    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" event={"ID":"72e0980a-4a4a-49cc-9627-07c8f6f460cb","Type":"ContainerStarted","Data":"b7227e807a2b35248578c40148c570a6faaf842d26461c8fca07ff4e7f4b1045"}
Oct 31 04:38:17.781559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.781535    6931 generic.go:356] "Generic (PLEG): container finished" podID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerID="c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b" exitCode=0
Oct 31 04:38:17.781675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.781573    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"
Oct 31 04:38:17.781675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.781611    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" event={"ID":"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d","Type":"ContainerDied","Data":"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"}
Oct 31 04:38:17.781675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.781657    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx" event={"ID":"2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d","Type":"ContainerDied","Data":"16f8322236ed17b8a59dc3a1709f96511f816d92790858a93341d2fe39ba277b"}
Oct 31 04:38:17.781809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.781680    6931 scope.go:122] "RemoveContainer" containerID="c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"
Oct 31 04:38:17.792834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.792671    6931 scope.go:122] "RemoveContainer" containerID="c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"
Oct 31 04:38:17.792982 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:17.792960    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b\": container with ID starting with c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b not found: ID does not exist" containerID="c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"
Oct 31 04:38:17.793025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.792992    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b"} err="failed to get container status \"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b\": rpc error: code = NotFound desc = could not find container \"c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b\": container with ID starting with c6a3af4b6cf92ef92112f79d3e3d8b1395c0bdd1dcb027595e97e162f28a930b not found: ID does not exist"
Oct 31 04:38:17.828772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.828707    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podStartSLOduration=1.8286905519999999 podStartE2EDuration="1.828690552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:17.825736591 +0000 UTC m=+1993.858736293" watchObservedRunningTime="2026-10-31 04:38:17.828690552 +0000 UTC m=+1993.861690231"
Oct 31 04:38:17.864884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.864849    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:38:17.871736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:17.871708    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-9tntx"]
Oct 31 04:38:18.122886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:18.122805    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" path="/var/lib/kubelet/pods/2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d/volumes"
Oct 31 04:38:18.184318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:18.184286    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:18.185644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:18.185616    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:18.185793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:18.185671    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:18.681583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:18.681532    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:19.183665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:19.183632    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:19.184095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:19.183687    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:20.184108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:20.184073    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:20.184548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:20.184170    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:20.482301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:20.480908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:21.184488 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:21.184448    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:21.184891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:21.184515    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:22.184329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:22.184297    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:22.184497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:22.184361    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:22.304177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:22.304055    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:23.183726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:23.183687    6931 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 04:38:23.183921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:23.183741    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 04:38:23.699185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:23.696852    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:24.189038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:24.188994    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:24.189469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:24.189438    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:24.190251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:24.190230    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:38:24.291639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:24.291598    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:26.120535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:26.120500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:26.785783 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:26.785747    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:38:26.786006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:26.785979    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy" containerID="cri-o://9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178" gracePeriod=30
Oct 31 04:38:28.151541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.151492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:28.259265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.259227    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7108-c55018"]
Oct 31 04:38:28.260229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.260206    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy"
Oct 31 04:38:28.260323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.260231    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy"
Oct 31 04:38:28.260412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.260400    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f42f8b8-5ff2-4c78-99cb-fab4b0c0fa1d" containerName="istio-proxy"
Oct 31 04:38:28.264497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.264478    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:28.279588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.279561    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-c55018"]
Oct 31 04:38:28.287623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.287571    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\""
Oct 31 04:38:28.289602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.289575    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkk2p\" (UniqueName: \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\") pod \"test-curl-7108-c55018\" (UID: \"385fd636-c31f-4e7a-8dbd-1271e42593b8\") " pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:28.404411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.404303    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkk2p\" (UniqueName: \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\") pod \"test-curl-7108-c55018\" (UID: \"385fd636-c31f-4e7a-8dbd-1271e42593b8\") " pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:28.432684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.429929    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkk2p\" (UniqueName: \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\") pod \"test-curl-7108-c55018\" (UID: \"385fd636-c31f-4e7a-8dbd-1271e42593b8\") " pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:28.447824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.447771    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:28.580312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.580280    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:28.885392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.885359    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7108-c55018"]
Oct 31 04:38:28.888863 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:38:28.888833    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod385fd636_c31f_4e7a_8dbd_1271e42593b8.slice/crio-15618d5a0c395df178547101ed9852d35723bc1b1a4b30b154d2c13fb4396342 WatchSource:0}: Error finding container 15618d5a0c395df178547101ed9852d35723bc1b1a4b30b154d2c13fb4396342: Status 404 returned error can't find the container with id 15618d5a0c395df178547101ed9852d35723bc1b1a4b30b154d2c13fb4396342
Oct 31 04:38:28.890700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:28.890681    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:38:29.871186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:29.871137    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-c55018" event={"ID":"385fd636-c31f-4e7a-8dbd-1271e42593b8","Type":"ContainerStarted","Data":"15618d5a0c395df178547101ed9852d35723bc1b1a4b30b154d2c13fb4396342"}
Oct 31 04:38:29.969666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:29.969636    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 04:38:29.973904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:29.973883    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:29.990369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:29.990336    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-766728-serving-cert\""
Oct 31 04:38:29.993950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:29.993925    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 04:38:30.024862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.024826    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.025030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.024907    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5pr\" (UniqueName: \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.025078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.025024    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.025127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.025110    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.126518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.126435    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.126703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.126589    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.126703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.126630    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5pr\" (UniqueName: \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.126703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.126692    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.127199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.127141    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.129301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.129281    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.129406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.129337    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.150519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.150480    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5pr\" (UniqueName: \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\") pod \"maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.169173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.169122    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:30.289557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.289515    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:30.399504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.399411    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:30.603295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.603257    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 04:38:30.606595 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:38:30.606566    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390db20a_4526_419e_9304_7f67e47cbe83.slice/crio-177414ee3a9dc5d9daac7d82a2b8b286bcbf251e9d126d85cc027d16b1040b88 WatchSource:0}: Error finding container 177414ee3a9dc5d9daac7d82a2b8b286bcbf251e9d126d85cc027d16b1040b88: Status 404 returned error can't find the container with id 177414ee3a9dc5d9daac7d82a2b8b286bcbf251e9d126d85cc027d16b1040b88
Oct 31 04:38:30.879073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.879040    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7108-c55018" event={"ID":"385fd636-c31f-4e7a-8dbd-1271e42593b8","Type":"ContainerStarted","Data":"55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef"}
Oct 31 04:38:30.880472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:30.880446    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" event={"ID":"390db20a-4526-419e-9304-7f67e47cbe83","Type":"ContainerStarted","Data":"177414ee3a9dc5d9daac7d82a2b8b286bcbf251e9d126d85cc027d16b1040b88"}
Oct 31 04:38:31.888815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.888779    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" event={"ID":"390db20a-4526-419e-9304-7f67e47cbe83","Type":"ContainerStarted","Data":"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"}
Oct 31 04:38:31.889313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.888863    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:31.917282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.917144    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7108-c55018" podStartSLOduration=2.933648333 podStartE2EDuration="3.917123862s" totalImagesPullingTime="983.475529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:30.928122071 +0000 UTC m=+2006.961121751" watchObservedRunningTime="2026-10-31 04:38:31.917123862 +0000 UTC m=+2007.950123542"
Oct 31 04:38:31.920763 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.920727    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7108-c55018"]
Oct 31 04:38:31.924919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.924883    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7108-c55018"]
Oct 31 04:38:31.932921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:31.932870    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" podStartSLOduration=2.43540762 podStartE2EDuration="2.932856277s" totalImagesPullingTime="497.448657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:38:31.930798163 +0000 UTC m=+2007.963797840" watchObservedRunningTime="2026-10-31 04:38:31.932856277 +0000 UTC m=+2007.965855981"
Oct 31 04:38:32.101667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.101632    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:38:32.121883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.121822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:32.178650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178614    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.178806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178722    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.178871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178812    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.178929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178876    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.178987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178923    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.178987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.178975    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.179094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179009    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.179094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179059    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.179204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179130    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:32.179204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179138    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\") pod \"87551beb-c484-4cf8-a770-5d9642f6d928\" (UID: \"87551beb-c484-4cf8-a770-5d9642f6d928\") "
Oct 31 04:38:32.179370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179328    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:32.179423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179381    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:32.179831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179692    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:32.179831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179784    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.179831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179797    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.179831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179805    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.179831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179814    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.180063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.179825    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:38:32.181758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.181723    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:32.181865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.181762    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:32.181992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.181972    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:38:32.182558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.182538    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg" pod "87551beb-c484-4cf8-a770-5d9642f6d928" (UID: "87551beb-c484-4cf8-a770-5d9642f6d928"). InnerVolumeSpecName "kube-api-access-cj2zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:32.280780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.280738    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.280780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.280768    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2zg\" (UniqueName: \"kubernetes.io/projected/87551beb-c484-4cf8-a770-5d9642f6d928-kube-api-access-cj2zg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.280780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.280784    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87551beb-c484-4cf8-a770-5d9642f6d928-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.281094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.280794    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87551beb-c484-4cf8-a770-5d9642f6d928-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.281094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.280803    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87551beb-c484-4cf8-a770-5d9642f6d928-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:32.897041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897003    6931 generic.go:356] "Generic (PLEG): container finished" podID="87551beb-c484-4cf8-a770-5d9642f6d928" containerID="9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178" exitCode=0
Oct 31 04:38:32.897479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897050    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"
Oct 31 04:38:32.897479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897088    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" event={"ID":"87551beb-c484-4cf8-a770-5d9642f6d928","Type":"ContainerDied","Data":"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"}
Oct 31 04:38:32.897479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897144    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf" event={"ID":"87551beb-c484-4cf8-a770-5d9642f6d928","Type":"ContainerDied","Data":"62bb92364daefa46425280776353c8b8f4a5c2031757aa77acee244e11ed81ca"}
Oct 31 04:38:32.897479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897208    6931 scope.go:122] "RemoveContainer" containerID="9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"
Oct 31 04:38:32.897479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.897410    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7108-c55018" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" containerName="test-curl-7108-c55018" containerID="cri-o://55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef" gracePeriod=2
Oct 31 04:38:32.908269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.908239    6931 scope.go:122] "RemoveContainer" containerID="9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"
Oct 31 04:38:32.908556 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:32.908537    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178\": container with ID starting with 9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178 not found: ID does not exist" containerID="9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"
Oct 31 04:38:32.908611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.908565    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178"} err="failed to get container status \"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178\": rpc error: code = NotFound desc = could not find container \"9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178\": container with ID starting with 9a2134f93fe8c17b5063dec1f198c67502f8e173a7204e9b4593775db82bc178 not found: ID does not exist"
Oct 31 04:38:32.970655 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.970611    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:38:32.982246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:32.982202    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-rvcxf"]
Oct 31 04:38:33.978268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:33.978231    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:34.125916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:34.125882    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" path="/var/lib/kubelet/pods/87551beb-c484-4cf8-a770-5d9642f6d928/volumes"
Oct 31 04:38:35.109885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.109860    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:35.225169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.225078    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\" (UniqueName: \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\") pod \"385fd636-c31f-4e7a-8dbd-1271e42593b8\" (UID: \"385fd636-c31f-4e7a-8dbd-1271e42593b8\") "
Oct 31 04:38:35.227272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.227215    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p" pod "385fd636-c31f-4e7a-8dbd-1271e42593b8" (UID: "385fd636-c31f-4e7a-8dbd-1271e42593b8"). InnerVolumeSpecName "kube-api-access-pkk2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:35.327114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.327064    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkk2p\" (UniqueName: \"kubernetes.io/projected/385fd636-c31f-4e7a-8dbd-1271e42593b8-kube-api-access-pkk2p\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:38:35.672516 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.672476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:35.907874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.907831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:35.918094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.918054    6931 generic.go:356] "Generic (PLEG): container finished" podID="385fd636-c31f-4e7a-8dbd-1271e42593b8" containerID="55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef" exitCode=137
Oct 31 04:38:35.918282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.918111    6931 scope.go:122] "RemoveContainer" containerID="55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef"
Oct 31 04:38:35.918282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.918117    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:35.929238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.929216    6931 scope.go:122] "RemoveContainer" containerID="55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef"
Oct 31 04:38:35.929549 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:35.929527    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef\": container with ID starting with 55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef not found: ID does not exist" containerID="55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef"
Oct 31 04:38:35.929640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:35.929554    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef"} err="failed to get container status \"55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef\": rpc error: code = NotFound desc = could not find container \"55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef\": container with ID starting with 55b7ffca4543d1b7c95628a8f1dfe00bb739a009911ccb8c72bc576a52b8c4ef not found: ID does not exist"
Oct 31 04:38:35.940794 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:35.940737    6931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7108-c55018\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" pod="opendatahub/test-curl-7108-c55018"
Oct 31 04:38:36.122903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:36.122869    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" path="/var/lib/kubelet/pods/385fd636-c31f-4e7a-8dbd-1271e42593b8/volumes"
Oct 31 04:38:37.702297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:37.702252    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:37.903481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:37.903452    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 04:38:39.599500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:39.599460    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:40.511720 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:40.511685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:40.761562 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:38:40.761526    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:38:40.984918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:40.984879    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:41.199185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:41.199141    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:41.676556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:41.676514    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:43.580511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:43.580472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:44.407888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:44.407844    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:44.632212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:44.628222    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:45.512192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:45.510736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:46.340771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:46.340729    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:47.839111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:47.839069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:48.195464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:48.195288    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:48.659296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:48.658647    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:50.486311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:50.486273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:50.740943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:50.740845    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:52.310199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:52.310119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:52.534089 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:52.534055    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:52.839394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:52.839323    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:53.053748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:53.053714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:54.356519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:54.356485    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:56.350869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:56.350829    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:56.893085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:56.893044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:57.111400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:57.111366    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:58.442071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:58.442032    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:59.684920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:38:59.684883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:00.480388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:00.480351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:02.341710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:02.341667    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:02.649897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:02.649801    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:04.221721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:04.221685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:04.449796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:04.449753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:04.710870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:04.710822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:05.144544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:05.144506    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:05.355473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:05.355432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:05.732555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:05.732511    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:06.147581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:06.147540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:07.404668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:07.404631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:08.179437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:08.179400    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:08.401364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:08.401319    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:10.001957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:10.001916    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:10.226142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:10.226093    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:10.440313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:10.440273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:10.759920 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:39:10.759837    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:39:11.952350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:11.952299    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:12.198116 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:12.196977    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:12.455573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:12.455535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:14.310180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:14.304808    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:14.620264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:14.620144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:14.844981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:14.844947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:16.245265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:16.245231    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:18.192233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:18.192186    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:20.008920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:20.008883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:21.741481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:21.741445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:23.694817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:23.694781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:25.488049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:25.488011    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:27.379616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:27.379578    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:27.592312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:27.592253    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:28.048752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:28.048715    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:39:28.048963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:28.048923    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" containerID="cri-o://5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6" gracePeriod=30
Oct 31 04:39:29.399998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:29.399953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:31.199988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:31.199943    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:32.633451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:32.633407    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:39:32.633885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:32.633469    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:39:33.013053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.012957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:33.332868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.332832    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:39:33.338831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.338807    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.338938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.338876    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.338977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.338944    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.338978    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339005    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339120 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339029    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339120 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339061    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339202    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339234    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\") pod \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\" (UID: \"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a\") "
Oct 31 04:39:33.339358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339304    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:39:33.339358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339321    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:39:33.339358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339331    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:39:33.339617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339596    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:39:33.339746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339721    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:39:33.339809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339737    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.339809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339773    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.339809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339792    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.339809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.339807    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.341345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.341321    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:39:33.341435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.341415    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "kube-api-access-mgwc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:33.341541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.341524    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:39:33.341583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.341540    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token" pod "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" (UID: "532f15ed-92bd-4c7c-92f9-bb0ffc3af80a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:33.441216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.441177    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.441216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.441214    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.441422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.441229    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.441422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.441243    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwc7\" (UniqueName: \"kubernetes.io/projected/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-kube-api-access-mgwc7\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.441422 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.441257    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:39:33.515839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.515801    6931 generic.go:356] "Generic (PLEG): container finished" podID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerID="5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6" exitCode=0
Oct 31 04:39:33.516034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.515851    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"
Oct 31 04:39:33.516034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.515888    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" event={"ID":"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a","Type":"ContainerDied","Data":"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"}
Oct 31 04:39:33.516034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.515940    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk" event={"ID":"532f15ed-92bd-4c7c-92f9-bb0ffc3af80a","Type":"ContainerDied","Data":"1632445bd1c21760a6751519d5aefc1f25695d7c544438560552899c04dac267"}
Oct 31 04:39:33.516034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.515963    6931 scope.go:122] "RemoveContainer" containerID="5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"
Oct 31 04:39:33.526712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.526695    6931 scope.go:122] "RemoveContainer" containerID="5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"
Oct 31 04:39:33.526964 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:39:33.526944    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6\": container with ID starting with 5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6 not found: ID does not exist" containerID="5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"
Oct 31 04:39:33.527031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.526974    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6"} err="failed to get container status \"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6\": rpc error: code = NotFound desc = could not find container \"5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6\": container with ID starting with 5108332d801cef3debde99574cdce56006f44a4a522d88e40f489a4522fbb4d6 not found: ID does not exist"
Oct 31 04:39:33.567788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.567700    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:39:33.571991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:33.571959    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zw5nk"]
Oct 31 04:39:34.123442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:34.123413    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" path="/var/lib/kubelet/pods/532f15ed-92bd-4c7c-92f9-bb0ffc3af80a/volumes"
Oct 31 04:39:34.819113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:34.819079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:36.612243 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:36.612210    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:38.341861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:38.341814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:40.110148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:40.110104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:40.323538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:40.323500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:40.761239 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:39:40.761197    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:39:41.991970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:41.991932    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:43.990182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:43.986814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:44.201962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:44.201891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:44.417435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:44.417396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:45.898713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:45.898656    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:47.710332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:47.710279    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:49.528068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:49.528026    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:50.991427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:50.991393    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:51.397949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:51.397907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:53.658971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:53.658937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:55.726397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:55.726356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:55.973416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:55.973373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:57.280391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:57.280348    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:57.489405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:57.489364    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:57.819182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:57.818638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:58.034878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:58.034838    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:58.255000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:58.254925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:58.563343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:58.562977    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:58.922967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:58.922882    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:59.410526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:39:59.410490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:00.214549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:00.214441    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:02.177652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:02.177603    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:02.430705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:02.430619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:02.648436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:02.648399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:03.883012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:03.882979    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:04.105619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:04.105585    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:04.475112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:04.475074    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:05.612660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:05.612613    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:05.904264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:05.902587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:06.978813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:06.978777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:08.271975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:08.271942    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:09.088536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:09.088500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:09.565218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:09.565180    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:10.042245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:10.035115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:10.471588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:10.471489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:10.760367 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:40:10.760287    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:40:11.098100 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:11.098057    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:12.904951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:12.904891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:14.299603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:14.299569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:14.513971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:14.513930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:14.946119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:14.946084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:16.006965 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:16.006918    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:16.225209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:16.225103    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:16.912486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:16.912443    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:18.774905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:18.774867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:20.736390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:20.736351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:22.647139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:22.647070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:24.794966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:24.794913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:25.013645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:25.013603    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:26.142608 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:26.142569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:26.618451 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:26.618402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:26.848133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:26.848069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:27.091000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:27.090962    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:27.315262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:27.315214    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:29.072445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:29.072408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:29.300236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:29.300194    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:29.554881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:29.554841    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:29.779584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:29.779543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:30.596625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:30.596592    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:30.826059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:30.826024    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:31.245784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:31.245743    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:32.211514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:32.211475    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:32.434349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:32.434297    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:32.949532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:32.949478    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:33.538485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:33.538448    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:33.749021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:33.748980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:35.362025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:35.361991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:35.655113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:35.653265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:35.875292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:35.875216    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:37.466429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:37.466396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:39.337213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:39.337129    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:40.759543 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:40:40.759509    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:40:40.958394 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958358    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:40:40.958953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958940    6931 cpu_manager.go:414] "containerMap: removing container" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy"
Oct 31 04:40:40.959003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958954    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy"
Oct 31 04:40:40.959003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958973    6931 cpu_manager.go:414] "containerMap: removing container" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" containerName="test-curl-7108-c55018"
Oct 31 04:40:40.959003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958978    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" containerName="test-curl-7108-c55018"
Oct 31 04:40:40.959003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958988    6931 cpu_manager.go:414] "containerMap: removing container" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy"
Oct 31 04:40:40.959003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.958993    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy"
Oct 31 04:40:40.959184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.959091    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="532f15ed-92bd-4c7c-92f9-bb0ffc3af80a" containerName="istio-proxy"
Oct 31 04:40:40.959184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.959100    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="385fd636-c31f-4e7a-8dbd-1271e42593b8" containerName="test-curl-7108-c55018"
Oct 31 04:40:40.959184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.959108    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87551beb-c484-4cf8-a770-5d9642f6d928" containerName="istio-proxy"
Oct 31 04:40:40.962496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.962479    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:40.979602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.979567    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-99ca0c-serving-cert\""
Oct 31 04:40:40.996845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:40.996808    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:40:41.056922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.056888    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.057075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.057036    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbqp\" (UniqueName: \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.057139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.057090    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.057206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.057145    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.158393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.158351    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbqp\" (UniqueName: \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.158595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.158402    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.158595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.158483    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.158693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.158615    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.159639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.159607    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.160295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.160266    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:41.161336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.161307    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.161443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.161357    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.181465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.181440    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbqp\" (UniqueName: \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\") pod \"maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.276188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.276143    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:41.373392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.373308    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:41.592073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:41.592034    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:40:41.595469 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:40:41.595441    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de12c3a_74c8_47cb_958c_8bca741ad99a.slice/crio-c62c9bea3f40fa93966edcc710996493d950304f43a030a8e2b59ed4c5916f3c WatchSource:0}: Error finding container c62c9bea3f40fa93966edcc710996493d950304f43a030a8e2b59ed4c5916f3c: Status 404 returned error can't find the container with id c62c9bea3f40fa93966edcc710996493d950304f43a030a8e2b59ed4c5916f3c
Oct 31 04:40:42.144829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.144790    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" event={"ID":"1de12c3a-74c8-47cb-958c-8bca741ad99a","Type":"ContainerStarted","Data":"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"}
Oct 31 04:40:42.145321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.144848    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" event={"ID":"1de12c3a-74c8-47cb-958c-8bca741ad99a","Type":"ContainerStarted","Data":"c62c9bea3f40fa93966edcc710996493d950304f43a030a8e2b59ed4c5916f3c"}
Oct 31 04:40:42.145321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.144879    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:42.187719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.187654    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" podStartSLOduration=1.777435117 podStartE2EDuration="2.187637872s" totalImagesPullingTime="410.202755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:42.185692272 +0000 UTC m=+2138.218691953" watchObservedRunningTime="2026-10-31 04:40:42.187637872 +0000 UTC m=+2138.220637552"
Oct 31 04:40:42.850654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.850572    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 04:40:42.854800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.854782    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.870878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.870846    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ded435-serving-cert\""
Oct 31 04:40:42.887799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.885877    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.887799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.885939    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.887799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.886068    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfxv\" (UniqueName: \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.887799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.886192    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.888205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.888175    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 04:40:42.987765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.987731    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfxv\" (UniqueName: \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.987956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.987813    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.987956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.987869    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.987956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.987896    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.988134 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:40:42.987992    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-serving-cert: secret "maas-api-e2e-extmod-b-ded435-serving-cert" not found
Oct 31 04:40:42.988134 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:40:42.988082    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls podName:a163d702-ad24-4230-9ee7-f1a42e40ec22 nodeName:}" failed. No retries permitted until 2026-10-31 04:40:43.488060661 +0000 UTC m=+2139.521060329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls") pod "maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" (UID: "a163d702-ad24-4230-9ee7-f1a42e40ec22") : secret "maas-api-e2e-extmod-b-ded435-serving-cert" not found
Oct 31 04:40:42.988650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.988632    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:42.990460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:42.990439    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:43.011076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.011041    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfxv\" (UniqueName: \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:43.045624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.045589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:43.358604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.358565    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:43.493938 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.493899    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:43.496363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.496341    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:43.769719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:43.769631    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:44.132378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:44.131715    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 04:40:44.186056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:44.186021    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" event={"ID":"a163d702-ad24-4230-9ee7-f1a42e40ec22","Type":"ContainerStarted","Data":"a095a21d9896ddd55cf78c2712286b77f9a27fedaa6f8264df56fe32a5a92203"}
Oct 31 04:40:45.015951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:45.015905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:45.195120 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:45.195070    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" event={"ID":"a163d702-ad24-4230-9ee7-f1a42e40ec22","Type":"ContainerStarted","Data":"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"}
Oct 31 04:40:45.195336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:45.195281    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:45.238199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:45.238115    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" podStartSLOduration=2.921760207 podStartE2EDuration="3.238096528s" totalImagesPullingTime="316.336321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:40:45.235889096 +0000 UTC m=+2141.268888785" watchObservedRunningTime="2026-10-31 04:40:45.238096528 +0000 UTC m=+2141.271096208"
Oct 31 04:40:46.919172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:46.919118    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:48.179672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:48.179642    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:40:48.643761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:48.643714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:50.424486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:50.424449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:51.207576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:51.207545    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 04:40:52.303185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:52.303115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:54.169718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:54.169681    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:55.874428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:55.874385    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:56.101329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:56.101296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:57.720386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:57.720351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:59.574022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:59.569139    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:59.783592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:40:59.783557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:01.326774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:01.326741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:03.111946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:03.111912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:04.916450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:04.916417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:06.773944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:06.773902    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:08.521770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:08.521735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:10.320141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:10.320102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:10.760375 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:41:10.760278    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:41:12.188721 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:12.188669    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:12.404286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:12.404239    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:12.635496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:12.635462    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:14.202581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:14.202479    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:15.795108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.795069    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 04:41:15.800382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.800361    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:15.832819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.832779    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 04:41:15.833014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.832899    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8973e2-serving-cert\""
Oct 31 04:41:15.956026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.955992    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ht6\" (UniqueName: \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:15.956250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.956034    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:15.956250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.956106    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:15.956250 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.956127    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:15.988397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:15.988352    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:16.057670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.057576    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.057670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.057617    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.057913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.057782    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ht6\" (UniqueName: \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.057913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.057806    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.058388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.058360    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.060222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.060200    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.060322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.060282    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.081674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.081644    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ht6\" (UniqueName: \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\") pod \"maas-api-e2e-extmod-a-8973e2-7b76888646-prj62\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.114568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.114539    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:16.219192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.214271    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:16.416814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.416634    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 04:41:16.422965 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:41:16.422926    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c31e963_b71e_4edb_a266_0baf09315cd6.slice/crio-7cf30e46582837df26889690b429620c36dde3c83c094cd9835f3de04cb9bde4 WatchSource:0}: Error finding container 7cf30e46582837df26889690b429620c36dde3c83c094cd9835f3de04cb9bde4: Status 404 returned error can't find the container with id 7cf30e46582837df26889690b429620c36dde3c83c094cd9835f3de04cb9bde4
Oct 31 04:41:16.437584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:16.437556    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:17.419507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:17.419460    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" event={"ID":"4c31e963-b71e-4edb-a266-0baf09315cd6","Type":"ContainerStarted","Data":"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"}
Oct 31 04:41:17.420003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:17.419521    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" event={"ID":"4c31e963-b71e-4edb-a266-0baf09315cd6","Type":"ContainerStarted","Data":"7cf30e46582837df26889690b429620c36dde3c83c094cd9835f3de04cb9bde4"}
Oct 31 04:41:17.420003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:17.419688    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:17.478336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:17.478267    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" podStartSLOduration=2.069659212 podStartE2EDuration="2.478247102s" totalImagesPullingTime="408.58789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:41:17.476322507 +0000 UTC m=+2173.509322185" watchObservedRunningTime="2026-10-31 04:41:17.478247102 +0000 UTC m=+2173.511246782"
Oct 31 04:41:18.064903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:18.064840    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:19.978349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:19.978307    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:21.770945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:21.770909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:23.431546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:23.431517    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 04:41:23.630219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:23.630175    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:25.418514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:25.418476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:27.259781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:27.259737    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:29.003485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:29.003442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:29.311457 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:29.311396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:31.275202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:31.275132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:31.484463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:31.484431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:33.225967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:33.225924    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:34.236956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:34.236907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:35.036791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:35.036717    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:35.439979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:35.439881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:35.659927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:35.659891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:35.886542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:35.886499    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:36.105436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:36.105400    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:36.328436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:36.328392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:36.842783 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:36.842748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:37.054746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:37.054711    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:38.654130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:38.654094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:38.863596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:38.863563    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:40.558050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:40.558017    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:40.760070 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:41:40.760029    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:41:42.506105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:42.506064    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:44.317692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:44.317657    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:46.178265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:46.178215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:46.442566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:46.442489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:47.355534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:47.355494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:47.583530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:47.583495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:47.792572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:47.792515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:48.011366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:48.011327    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:48.250237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:48.247335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:48.767767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:48.767727    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:48.979838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:48.979804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:50.250966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:50.250934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:50.535275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:50.532710    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:51.342847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:51.342813    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:51.879246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:51.879209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:52.125231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:52.125170    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:53.890595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:53.890506    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:55.689564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:55.689519    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:56.985256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:56.985190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:57.491281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:57.491235    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:59.427232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:59.427197    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:59.830019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:41:59.829988    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:01.305679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:01.305647    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:02.489329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:02.489280    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:03.327810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:03.327766    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:03.669140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:03.669029    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:03.882687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:03.882655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:04.098058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:04.098013    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:04.327172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:04.327115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:04.545711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:04.545672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:05.026988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:05.026906    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:05.266177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:05.265707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:05.491784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:05.491749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:05.704737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:05.704686    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:06.430179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:06.428748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:06.927133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:06.927095    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:42:06.927367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:06.927326    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy" containerID="cri-o://d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d" gracePeriod=30
Oct 31 04:42:06.985667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:06.985632    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:07.456902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:07.456861    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:07.683977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:07.683942    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:09.316698 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:09.316655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:10.865388 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:42:10.865338    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:42:11.283686 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:11.283659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:11.736618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:11.736524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:11.965110 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:11.965077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:12.207216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.207187    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:42:12.358562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358479    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358526    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358551    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358703    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358801    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358841    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358875    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.358927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358883    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:12.359055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.358893    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.359055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359009    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\") pod \"b45588b4-b059-428e-977a-d432a1f27403\" (UID: \"b45588b4-b059-428e-977a-d432a1f27403\") "
Oct 31 04:42:12.359203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359084    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:12.359487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359287    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:12.359487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359300    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:12.359487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359462    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:12.359726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359698    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.359726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359710    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.359726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359721    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.359890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359730    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.359890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.359738    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45588b4-b059-428e-977a-d432a1f27403-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.360991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.360968    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:12.361151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.361130    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:42:12.361324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.361303    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:12.361367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.361303    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8" pod "b45588b4-b059-428e-977a-d432a1f27403" (UID: "b45588b4-b059-428e-977a-d432a1f27403"). InnerVolumeSpecName "kube-api-access-6zfs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:12.460718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.460684    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45588b4-b059-428e-977a-d432a1f27403-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.460718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.460712    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45588b4-b059-428e-977a-d432a1f27403-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.460718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.460722    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.460955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.460732    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zfs8\" (UniqueName: \"kubernetes.io/projected/b45588b4-b059-428e-977a-d432a1f27403-kube-api-access-6zfs8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:12.525430 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.525394    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:42:12.525642 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.525596    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy" containerID="cri-o://8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698" gracePeriod=30
Oct 31 04:42:12.848382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.848344    6931 generic.go:356] "Generic (PLEG): container finished" podID="b45588b4-b059-428e-977a-d432a1f27403" containerID="d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d" exitCode=0
Oct 31 04:42:12.848558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.848400    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"
Oct 31 04:42:12.848558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.848434    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" event={"ID":"b45588b4-b059-428e-977a-d432a1f27403","Type":"ContainerDied","Data":"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"}
Oct 31 04:42:12.848558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.848496    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz" event={"ID":"b45588b4-b059-428e-977a-d432a1f27403","Type":"ContainerDied","Data":"a7ba200265c77d9a048a58e47a600bebd9e8d3416e22135cb8051b170c044e9c"}
Oct 31 04:42:12.848558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.848520    6931 scope.go:122] "RemoveContainer" containerID="d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"
Oct 31 04:42:12.859000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.858980    6931 scope.go:122] "RemoveContainer" containerID="d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"
Oct 31 04:42:12.859269 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:42:12.859251    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d\": container with ID starting with d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d not found: ID does not exist" containerID="d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"
Oct 31 04:42:12.859346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.859277    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d"} err="failed to get container status \"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d\": rpc error: code = NotFound desc = could not find container \"d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d\": container with ID starting with d45fba7580aa746e9491074b53f9a9afcc499a8bfbe1f1928b49ac5f4346184d not found: ID does not exist"
Oct 31 04:42:12.904182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.904140    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:42:12.919782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:12.919736    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-4l8fz"]
Oct 31 04:42:13.249780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:13.249694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:13.556334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:13.556294    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:13.805180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:13.790803    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:14.125086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:14.125050    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45588b4-b059-428e-977a-d432a1f27403" path="/var/lib/kubelet/pods/b45588b4-b059-428e-977a-d432a1f27403/volumes"
Oct 31 04:42:15.207587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:15.207553    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:16.913620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:16.913584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:17.792619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.792555    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:42:17.856024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.855988    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.857221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.857195    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:17.895121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.895083    6931 generic.go:356] "Generic (PLEG): container finished" podID="244dbe89-ede1-4972-8e08-185804450fa2" containerID="8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698" exitCode=0
Oct 31 04:42:17.895309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.895143    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"
Oct 31 04:42:17.895309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.895224    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" event={"ID":"244dbe89-ede1-4972-8e08-185804450fa2","Type":"ContainerDied","Data":"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"}
Oct 31 04:42:17.895452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.895324    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf" event={"ID":"244dbe89-ede1-4972-8e08-185804450fa2","Type":"ContainerDied","Data":"1f87f188cbf78c2db6d1d1b320f4c66a230f5948770c125ef980d3eaf0af9d5d"}
Oct 31 04:42:17.895452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.895359    6931 scope.go:122] "RemoveContainer" containerID="8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"
Oct 31 04:42:17.906503 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.906483    6931 scope.go:122] "RemoveContainer" containerID="8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"
Oct 31 04:42:17.906812 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:42:17.906791    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698\": container with ID starting with 8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698 not found: ID does not exist" containerID="8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"
Oct 31 04:42:17.906874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.906823    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698"} err="failed to get container status \"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698\": rpc error: code = NotFound desc = could not find container \"8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698\": container with ID starting with 8f20252e3093664ff5f058e6c5010a6475129e5be167fcb116e1c81c6e158698 not found: ID does not exist"
Oct 31 04:42:17.958676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.958626    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.959929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.959908    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.959833    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:17.960364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960338    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:17.960535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960412    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960474    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960551    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960590    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960714 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960641    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.960895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.960781    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\") pod \"244dbe89-ede1-4972-8e08-185804450fa2\" (UID: \"244dbe89-ede1-4972-8e08-185804450fa2\") "
Oct 31 04:42:17.961444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.961414    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:17.961821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.961792    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:17.961904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.961825    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dbe89-ede1-4972-8e08-185804450fa2-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:17.961962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.961841    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:17.961962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.961925    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:17.962114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.962095    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:17.963219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.963194    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:17.963386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.963365    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "kube-api-access-6df4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:17.963466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.963445    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:42:17.963466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:17.963454    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo" pod "244dbe89-ede1-4972-8e08-185804450fa2" (UID: "244dbe89-ede1-4972-8e08-185804450fa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:42:18.063759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.063659    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6df4d\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-kube-api-access-6df4d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:18.063759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.063707    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dbe89-ede1-4972-8e08-185804450fa2-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:18.063759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.063717    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:18.063759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.063726    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dbe89-ede1-4972-8e08-185804450fa2-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:18.063759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.063734    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dbe89-ede1-4972-8e08-185804450fa2-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:18.235022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.234982    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:42:18.246188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.246149    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-5dzkf"]
Oct 31 04:42:18.708393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:18.708352    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:20.122725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:20.122682    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244dbe89-ede1-4972-8e08-185804450fa2" path="/var/lib/kubelet/pods/244dbe89-ede1-4972-8e08-185804450fa2/volumes"
Oct 31 04:42:20.515013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:20.514916    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:22.305484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:22.305437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:24.033528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:24.033487    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:25.829375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:25.829244    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:27.616303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:27.616265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:29.339734 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:29.339691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:31.130532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:31.130493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:32.953654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:32.953603    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:33.376222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:33.376170    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:33.591987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:33.591947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:34.808102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:34.808061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:35.032147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:35.032104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:36.700276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:36.700235    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:38.504398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:38.504343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:40.258939 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:40.258903    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:40.567128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:40.567091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:40.759725 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:42:40.759680    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:42:41.984447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:41.984385    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:43.686666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:43.686631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:45.420085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:45.419946    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:47.208561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:47.208515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:48.939441 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:48.939402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:50.810719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:50.810679    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:52.714837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:52.714800    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:54.370467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.370433    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 04:42:54.371014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371000    6931 cpu_manager.go:414] "containerMap: removing container" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy"
Oct 31 04:42:54.371062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371015    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy"
Oct 31 04:42:54.371062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371051    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy"
Oct 31 04:42:54.371062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371056    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy"
Oct 31 04:42:54.371183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371174    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45588b4-b059-428e-977a-d432a1f27403" containerName="istio-proxy"
Oct 31 04:42:54.371221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.371189    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="244dbe89-ede1-4972-8e08-185804450fa2" containerName="istio-proxy"
Oct 31 04:42:54.374770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.374753    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.401924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.401891    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-227c93-serving-cert\""
Oct 31 04:42:54.405683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.405642    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 04:42:54.513833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.513800    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtdw\" (UniqueName: \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.513833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.513840    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.514075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.513881    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.514075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.514007    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.560818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.560775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:54.615822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.615781    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtdw\" (UniqueName: \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.616083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.616063    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.616271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.616253    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.616506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.616486    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.620676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.620598    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.625830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.625721    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.626867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.626819    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.640737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.640601    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtdw\" (UniqueName: \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\") pod \"maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.687963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.687926    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:54.787513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.787474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:54.995301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:54.995218    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 04:42:54.999376 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:42:54.999347    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955d9376_e518_4754_8434_175eb794b910.slice/crio-88c93e9992c2bdf8ae2c7cf58343f08cfbc13ae4994e6594c650e93b395aa2af WatchSource:0}: Error finding container 88c93e9992c2bdf8ae2c7cf58343f08cfbc13ae4994e6594c650e93b395aa2af: Status 404 returned error can't find the container with id 88c93e9992c2bdf8ae2c7cf58343f08cfbc13ae4994e6594c650e93b395aa2af
Oct 31 04:42:55.074191 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.072827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:55.142318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.142060    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" event={"ID":"955d9376-e518-4754-8434-175eb794b910","Type":"ContainerStarted","Data":"88c93e9992c2bdf8ae2c7cf58343f08cfbc13ae4994e6594c650e93b395aa2af"}
Oct 31 04:42:55.288458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.288419    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:55.527069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.527030    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:55.754235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.753768    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:55.966992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:55.966951    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:56.150923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.150886    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" event={"ID":"955d9376-e518-4754-8434-175eb794b910","Type":"ContainerStarted","Data":"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"}
Oct 31 04:42:56.151123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.151020    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:42:56.206129 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.206076    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:56.237604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.237549    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" podStartSLOduration=1.821346144 podStartE2EDuration="2.237531999s" totalImagesPullingTime="416.185855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:42:56.202608437 +0000 UTC m=+2272.235608142" watchObservedRunningTime="2026-10-31 04:42:56.237531999 +0000 UTC m=+2272.270531711"
Oct 31 04:42:56.237894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.237874    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:42:56.238558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.238525    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" podUID="1de12c3a-74c8-47cb-958c-8bca741ad99a" containerName="maas-api" containerID="cri-o://c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c" gracePeriod=30
Oct 31 04:42:56.454668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.454633    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:56.573175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.573135    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:42:56.674323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.674244    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\" (UniqueName: \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\") pod \"1de12c3a-74c8-47cb-958c-8bca741ad99a\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") "
Oct 31 04:42:56.674513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.674374    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\") pod \"1de12c3a-74c8-47cb-958c-8bca741ad99a\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") "
Oct 31 04:42:56.674513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.674418    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\") pod \"1de12c3a-74c8-47cb-958c-8bca741ad99a\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") "
Oct 31 04:42:56.674513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.674457    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\") pod \"1de12c3a-74c8-47cb-958c-8bca741ad99a\" (UID: \"1de12c3a-74c8-47cb-958c-8bca741ad99a\") "
Oct 31 04:42:56.674916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.674886    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca" pod "1de12c3a-74c8-47cb-958c-8bca741ad99a" (UID: "1de12c3a-74c8-47cb-958c-8bca741ad99a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:42:56.676489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.676460    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp" pod "1de12c3a-74c8-47cb-958c-8bca741ad99a" (UID: "1de12c3a-74c8-47cb-958c-8bca741ad99a"). InnerVolumeSpecName "kube-api-access-qwbqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:42:56.676589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.676571    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls" pod "1de12c3a-74c8-47cb-958c-8bca741ad99a" (UID: "1de12c3a-74c8-47cb-958c-8bca741ad99a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:42:56.676633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.676588    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs" pod "1de12c3a-74c8-47cb-958c-8bca741ad99a" (UID: "1de12c3a-74c8-47cb-958c-8bca741ad99a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:42:56.678313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.678285    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:56.776799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.776760    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:56.776799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.776797    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de12c3a-74c8-47cb-958c-8bca741ad99a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:56.777015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.776814    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwbqp\" (UniqueName: \"kubernetes.io/projected/1de12c3a-74c8-47cb-958c-8bca741ad99a-kube-api-access-qwbqp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:56.777015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.776831    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de12c3a-74c8-47cb-958c-8bca741ad99a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:42:56.929238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:56.923709    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:57.158964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.158931    6931 generic.go:356] "Generic (PLEG): container finished" podID="1de12c3a-74c8-47cb-958c-8bca741ad99a" containerID="c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c" exitCode=0
Oct 31 04:42:57.159202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.158980    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"
Oct 31 04:42:57.159202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.159012    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" event={"ID":"1de12c3a-74c8-47cb-958c-8bca741ad99a","Type":"ContainerDied","Data":"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"}
Oct 31 04:42:57.159202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.159061    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf" event={"ID":"1de12c3a-74c8-47cb-958c-8bca741ad99a","Type":"ContainerDied","Data":"c62c9bea3f40fa93966edcc710996493d950304f43a030a8e2b59ed4c5916f3c"}
Oct 31 04:42:57.159202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.159085    6931 scope.go:122] "RemoveContainer" containerID="c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"
Oct 31 04:42:57.170834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.170813    6931 scope.go:122] "RemoveContainer" containerID="c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"
Oct 31 04:42:57.171119 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:42:57.171098    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c\": container with ID starting with c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c not found: ID does not exist" containerID="c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"
Oct 31 04:42:57.171199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.171132    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c"} err="failed to get container status \"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c\": rpc error: code = NotFound desc = could not find container \"c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c\": container with ID starting with c5d7b44e46ba8802a1a2c47f05dbd31f3898196471701435573cb9760e2c551c not found: ID does not exist"
Oct 31 04:42:57.212918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.212826    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:42:57.245941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.244094    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-99ca0c-5956c8c67d-pqwjf"]
Oct 31 04:42:57.331531 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.331481    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:57.765179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.764787    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:57.975134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:57.975100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:58.124955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:58.124916    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de12c3a-74c8-47cb-958c-8bca741ad99a" path="/var/lib/kubelet/pods/1de12c3a-74c8-47cb-958c-8bca741ad99a/volumes"
Oct 31 04:42:58.194443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:58.194408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:59.003521 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:59.003484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:59.247365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:59.247328    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:59.952657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:42:59.952624    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:01.100311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:01.100273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:02.165027 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:02.164992    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 04:43:02.823267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:02.823228    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:04.626577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:04.626538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:05.797452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:05.797416    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:06.113070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:06.112980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:06.561543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:06.561504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:06.781128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:06.781089    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:07.136182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:07.134133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:08.506273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:08.506238    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:09.030177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:09.030117    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:09.529233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:09.515141    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:09.777826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:09.777791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:10.276518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:10.276470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:10.759384 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:43:10.759342    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:43:12.124747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:12.124711    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:13.784003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:13.783967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:13.997822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:13.997779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:14.208058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:14.207955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:15.504717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:15.504680    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:15.748180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:15.747624    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:16.080173 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:16.080128    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:17.996278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:17.996233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:19.718040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:19.718004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:21.493203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:21.493131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:23.307547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:23.307513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:25.034203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:25.034148    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:26.833380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:26.833343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:28.716147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:28.716101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:30.515464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:30.515426    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:31.334951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:31.334912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:31.550502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:31.550450    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:32.545171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:32.545119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:32.786219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:32.786148    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:33.058566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:33.058528    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:34.608047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:34.608012    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:36.398366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:36.398283    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:38.185735 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:38.185697    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:39.912653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:39.912616    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:40.760129 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:43:40.760094    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:43:41.747704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:41.747666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:43.556994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:43.556957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:43.792254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:43.792211    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:44.952291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:44.952251    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:45.172649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:45.172615    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:45.392555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:45.392513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:45.613839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:45.613802    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:45.826112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:45.826070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:46.434955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:46.434841    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:46.648372 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:46.648335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:47.324306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:47.324271    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:49.110544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:49.110505    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:50.904715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:50.904672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:52.291023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:52.290983    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:52.560771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:52.560678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:52.786520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:52.786484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:54.615652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:54.615605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:56.419103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:56.418982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:57.971542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.971494    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 04:43:57.972995 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.972966    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1de12c3a-74c8-47cb-958c-8bca741ad99a" containerName="maas-api"
Oct 31 04:43:57.972995 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.972995    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de12c3a-74c8-47cb-958c-8bca741ad99a" containerName="maas-api"
Oct 31 04:43:57.973334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.973316    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de12c3a-74c8-47cb-958c-8bca741ad99a" containerName="maas-api"
Oct 31 04:43:57.979978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.979950    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:57.986426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.986393    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:57.986560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.986503    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:57.986560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.986535    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84g6\" (UniqueName: \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:57.986679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.986644    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:57.994733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:57.994697    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 04:43:58.007490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.007453    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-47c409-serving-cert\""
Oct 31 04:43:58.087472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.087424    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.087680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.087578    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.087680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.087618    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84g6\" (UniqueName: \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.087809 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:43:58.087713    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-serving-cert: secret "maas-api-e2e-extmod-a-47c409-serving-cert" not found
Oct 31 04:43:58.087809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.087729    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.087809 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:43:58.087800    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls podName:98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0 nodeName:}" failed. No retries permitted until 2026-10-31 04:43:58.587783086 +0000 UTC m=+2334.620782747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls") pod "maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" (UID: "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0") : secret "maas-api-e2e-extmod-a-47c409-serving-cert" not found
Oct 31 04:43:58.088375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.088356    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.090061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.090040    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.113215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.113181    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84g6\" (UniqueName: \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.167795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.167752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:58.372741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.372685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:58.596742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.596698    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.600335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.600277    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.735554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.735464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:58.904349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.895712    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:43:58.951214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:58.951178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:59.249882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:59.249835    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 04:43:59.264765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:59.264740    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:43:59.358556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:59.358524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:59.631244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:43:59.631214    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" event={"ID":"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0","Type":"ContainerStarted","Data":"50547864f2b9591067550bfc6ea6b55b763952c027110b3cddfbcda30a1dd963"}
Oct 31 04:44:00.098189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:00.098143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:00.639092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:00.639053    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" event={"ID":"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0","Type":"ContainerStarted","Data":"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"}
Oct 31 04:44:00.639546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:00.639227    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:44:00.682961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:00.682907    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" podStartSLOduration=3.318385887 podStartE2EDuration="3.682889056s" totalImagesPullingTime="364.503169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:44:00.680521652 +0000 UTC m=+2336.713521331" watchObservedRunningTime="2026-10-31 04:44:00.682889056 +0000 UTC m=+2336.715888739"
Oct 31 04:44:01.852043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:01.851982    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:02.880088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:02.880055    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:44:02.880576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:02.880272    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy" containerID="cri-o://f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4" gracePeriod=30
Oct 31 04:44:02.891248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:02.891224    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:44:02.891464 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:02.891439    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" containerID="cri-o://bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb" gracePeriod=30
Oct 31 04:44:03.807336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:03.807289    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:05.630666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:05.630618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:06.328056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:06.328022    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:06.535043 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:06.535002    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:06.651199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:06.651114    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 04:44:07.240258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:07.240218    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:44:07.240442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:07.240279    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:44:07.422794 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:07.422753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:07.645863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:07.640906    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:08.365927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.365900    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:44:08.368859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.368840    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:44:08.514902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.514864    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.514920    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515003    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515061    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515123    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515185    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515220    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515256    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515286    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515315    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515325    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515371    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515429    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515482    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515529    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:08.515871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515572    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.515871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515627    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.515871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515629    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.516025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515864    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.516025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.515907    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.516137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516047    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.516234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516180    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.516366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516342    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:08.516575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516557    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.516772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516756    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\") pod \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\" (UID: \"0df059bf-ad76-46b7-b975-2a71cc4d2aed\") "
Oct 31 04:44:08.516892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.516877    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\") pod \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\" (UID: \"6dbac4ab-c725-48cb-a326-d17f6aeaa09b\") "
Oct 31 04:44:08.517656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.517633    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.517783 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.517769    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.517881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.517866    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.517985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.517973    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.518102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518088    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.518258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518234    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.518258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518258    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.518448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518273    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.518523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518503    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.519146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518765    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "kube-api-access-8g2dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:08.519146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.518834    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:08.519146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.519092    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.519146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.519109    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.519593 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.519569    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "kube-api-access-vq2x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:08.519718 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.519690    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:44:08.520005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.519980    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:08.520389 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.520366    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy" pod "0df059bf-ad76-46b7-b975-2a71cc4d2aed" (UID: "0df059bf-ad76-46b7-b975-2a71cc4d2aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:08.520622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.520604    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo" pod "6dbac4ab-c725-48cb-a326-d17f6aeaa09b" (UID: "6dbac4ab-c725-48cb-a326-d17f6aeaa09b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619331    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2dv\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-kube-api-access-8g2dv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619363    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619374    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619382    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619391    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619398    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619406    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619414    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619423    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq2x8\" (UniqueName: \"kubernetes.io/projected/6dbac4ab-c725-48cb-a326-d17f6aeaa09b-kube-api-access-vq2x8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.619802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.619430    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df059bf-ad76-46b7-b975-2a71cc4d2aed-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:08.762175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.762122    6931 generic.go:356] "Generic (PLEG): container finished" podID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerID="f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4" exitCode=0
Oct 31 04:44:08.762376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.762187    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"
Oct 31 04:44:08.762376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.762193    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" event={"ID":"6dbac4ab-c725-48cb-a326-d17f6aeaa09b","Type":"ContainerDied","Data":"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"}
Oct 31 04:44:08.762376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.762253    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk" event={"ID":"6dbac4ab-c725-48cb-a326-d17f6aeaa09b","Type":"ContainerDied","Data":"92f0186cce2709551f648b39359548e9a88b619cd8aeac63eb4b44a64536d7fc"}
Oct 31 04:44:08.762376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.762277    6931 scope.go:122] "RemoveContainer" containerID="f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"
Oct 31 04:44:08.763873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.763739    6931 generic.go:356] "Generic (PLEG): container finished" podID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerID="bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb" exitCode=0
Oct 31 04:44:08.763873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.763795    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"
Oct 31 04:44:08.763873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.763866    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" event={"ID":"0df059bf-ad76-46b7-b975-2a71cc4d2aed","Type":"ContainerDied","Data":"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"}
Oct 31 04:44:08.764050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.763902    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx" event={"ID":"0df059bf-ad76-46b7-b975-2a71cc4d2aed","Type":"ContainerDied","Data":"292e1d3fa66f175aee51f2a325b11ab675dab61d0b9431065b9aa1d3f9ec7611"}
Oct 31 04:44:08.774204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.774185    6931 scope.go:122] "RemoveContainer" containerID="f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"
Oct 31 04:44:08.774531 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:44:08.774509    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4\": container with ID starting with f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4 not found: ID does not exist" containerID="f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"
Oct 31 04:44:08.774620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.774539    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4"} err="failed to get container status \"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4\": rpc error: code = NotFound desc = could not find container \"f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4\": container with ID starting with f9b785d75b6a50b850ad4e227fb1a706472e03e642b09318834f271f221a2ee4 not found: ID does not exist"
Oct 31 04:44:08.774620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.774565    6931 scope.go:122] "RemoveContainer" containerID="bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"
Oct 31 04:44:08.784256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.784238    6931 scope.go:122] "RemoveContainer" containerID="bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"
Oct 31 04:44:08.784561 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:44:08.784543    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb\": container with ID starting with bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb not found: ID does not exist" containerID="bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"
Oct 31 04:44:08.784603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.784572    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb"} err="failed to get container status \"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb\": rpc error: code = NotFound desc = could not find container \"bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb\": container with ID starting with bfd0c29cf2926509e3dc08c1d0ceb66d2d82a242afa087d602f9683e4bcd50bb not found: ID does not exist"
Oct 31 04:44:08.815473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.815437    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:44:08.827304 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.827274    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-c7gdx"]
Oct 31 04:44:08.853264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.853228    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:44:08.861065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:08.861036    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-5qzmk"]
Oct 31 04:44:09.311902 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:09.311860    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:09.944179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:09.934827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:10.125230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:10.125193    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" path="/var/lib/kubelet/pods/0df059bf-ad76-46b7-b975-2a71cc4d2aed/volumes"
Oct 31 04:44:10.125922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:10.125901    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" path="/var/lib/kubelet/pods/6dbac4ab-c725-48cb-a326-d17f6aeaa09b/volumes"
Oct 31 04:44:10.759842 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:44:10.759809    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:44:11.208068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:11.208035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:12.911449 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:12.911411    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:13.129904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:13.129867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:13.357434 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:13.357374    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:13.575787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:13.575752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:13.793703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:13.793664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:14.365192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:14.365147    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:14.582188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:14.582100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:15.196659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:15.196617    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:15.730809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:15.730775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:16.484668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:16.484630    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:17.272490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:17.272453    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:17.923848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:17.923806    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:44:17.924079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:17.924056    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy" containerID="cri-o://cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b" gracePeriod=30
Oct 31 04:44:19.099241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:19.099195    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:20.823348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:20.823317    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:22.554936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:22.554897    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:23.199645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.199578    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:44:23.241768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241735    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.241915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241826    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.241915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241851    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.241915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241903    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241954    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.241992    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242004    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:23.242093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242060    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242224    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242259    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:23.242367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242292    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\") pod \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\" (UID: \"72e0980a-4a4a-49cc-9627-07c8f6f460cb\") "
Oct 31 04:44:23.242367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242304    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:44:23.242582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242421    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:23.242637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242609    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:23.242770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242752    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.242837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242771    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.242837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242780    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.242837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242789    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.242837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.242802    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.244402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.244379    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "kube-api-access-m8k4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:23.244402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.244392    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:44:23.244565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.244520    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:44:23.244644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.244627    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy" pod "72e0980a-4a4a-49cc-9627-07c8f6f460cb" (UID: "72e0980a-4a4a-49cc-9627-07c8f6f460cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:44:23.344247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.344205    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.344247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.344239    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8k4g\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-kube-api-access-m8k4g\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.344247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.344251    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.344481 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.344259    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e0980a-4a4a-49cc-9627-07c8f6f460cb-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:44:23.877861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.877828    6931 generic.go:356] "Generic (PLEG): container finished" podID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerID="cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b" exitCode=0
Oct 31 04:44:23.878293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.877878    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"
Oct 31 04:44:23.878293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.877911    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" event={"ID":"72e0980a-4a4a-49cc-9627-07c8f6f460cb","Type":"ContainerDied","Data":"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"}
Oct 31 04:44:23.878293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.877986    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt" event={"ID":"72e0980a-4a4a-49cc-9627-07c8f6f460cb","Type":"ContainerDied","Data":"b7227e807a2b35248578c40148c570a6faaf842d26461c8fca07ff4e7f4b1045"}
Oct 31 04:44:23.878293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.878008    6931 scope.go:122] "RemoveContainer" containerID="cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"
Oct 31 04:44:23.888262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.888243    6931 scope.go:122] "RemoveContainer" containerID="cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"
Oct 31 04:44:23.888513 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:44:23.888496    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b\": container with ID starting with cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b not found: ID does not exist" containerID="cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"
Oct 31 04:44:23.888557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.888522    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b"} err="failed to get container status \"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b\": rpc error: code = NotFound desc = could not find container \"cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b\": container with ID starting with cf66dbf495ca33bb952d99547fee7219b866365230e0369475dc55875238350b not found: ID does not exist"
Oct 31 04:44:23.942553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.942520    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:44:23.955748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:23.955716    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2a372-openshift-default-b75db95f7-f2cvt"]
Oct 31 04:44:24.123222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:24.123189    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" path="/var/lib/kubelet/pods/72e0980a-4a4a-49cc-9627-07c8f6f460cb/volumes"
Oct 31 04:44:24.314432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:24.314398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:26.054623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:26.054581    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:27.854379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:27.854332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:29.605461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:29.605424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:31.329091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:31.329056    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:33.115199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:33.115115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:34.922554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:34.922512    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:36.782893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:36.782843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:38.510770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:38.510732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:40.489629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:40.489596    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:40.702225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:40.702151    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:40.759384 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:44:40.759294    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:44:42.117407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:42.117370    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:42.329335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:42.329297    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:42.727221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:42.727182    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:44.235900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:44.235859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:45.953672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:45.953638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:47.784793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:47.784755    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:48.038306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:48.038224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:48.785181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:48.779608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:48.993344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:48.993299    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:49.299754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:49.299689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:49.747717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:49.747631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:50.941668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:50.941625    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:51.513392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:51.513356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:52.276775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:52.276737    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:52.741190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:52.740687    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:53.368934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:53.368893    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:53.582309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:53.582261    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:54.353761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:54.353704    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:54.564109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:54.564066    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:55.316853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:55.316816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:55.543485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:55.543448    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:55.755210 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:55.755087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:56.779545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:56.779508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:56.990673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:56.990634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:57.223005 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:57.222969    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:58.466888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.466859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:58.541619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.541584    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 04:44:58.542185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542151    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy"
Oct 31 04:44:58.542185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542179    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy"
Oct 31 04:44:58.542273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542208    6931 cpu_manager.go:414] "containerMap: removing container" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy"
Oct 31 04:44:58.542273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542213    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy"
Oct 31 04:44:58.542273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542239    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy"
Oct 31 04:44:58.542273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542245    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy"
Oct 31 04:44:58.542406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542341    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df059bf-ad76-46b7-b975-2a71cc4d2aed" containerName="istio-proxy"
Oct 31 04:44:58.542406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542354    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbac4ab-c725-48cb-a326-d17f6aeaa09b" containerName="istio-proxy"
Oct 31 04:44:58.542406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.542364    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e0980a-4a4a-49cc-9627-07c8f6f460cb" containerName="istio-proxy"
Oct 31 04:44:58.546198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.546181    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.564106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.564081    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 04:44:58.573079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.573036    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a04e4e\""
Oct 31 04:44:58.573227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.573122    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a04e4e-dockercfg-lfjd4\""
Oct 31 04:44:58.676188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.654954    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncc6m\" (UniqueName: \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.676500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.676470    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.697218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.658960    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 04:44:58.699875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.692797    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:58.718727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.718657    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.722196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.721181    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 04:44:58.778500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.778465    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.778695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.778520    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25vbj\" (UniqueName: \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.778695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.778648    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncc6m\" (UniqueName: \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.778695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.778689    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.779403 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.779385    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.801482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.801449    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncc6m\" (UniqueName: \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\") pod \"payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.859490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.859451    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:44:58.880416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.880375    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.880659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.880478    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25vbj\" (UniqueName: \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.881392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.881361    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:58.904054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:58.904000    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25vbj\" (UniqueName: \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\") pod \"payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:59.049084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:59.049046    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:44:59.122509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:59.122474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:59.165797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:59.165765    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 04:44:59.171069 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:44:59.171039    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e33b65_8e7f_4ce6_8199_8c60671a84e4.slice/crio-e419b90b1b7faff4b7b556477be171c14dd583e37b3d0100d4decb55628fb9c1 WatchSource:0}: Error finding container e419b90b1b7faff4b7b556477be171c14dd583e37b3d0100d4decb55628fb9c1: Status 404 returned error can't find the container with id e419b90b1b7faff4b7b556477be171c14dd583e37b3d0100d4decb55628fb9c1
Oct 31 04:44:59.389086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:44:59.388873    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 04:44:59.391918 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:44:59.391884    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82010fd1_72b3_46f4_9ec8_f9909ade03b8.slice/crio-cbcbef4e9f7c3993e5c0ab8af9d44966b271acded35fff9c90b4ac8be7638554 WatchSource:0}: Error finding container cbcbef4e9f7c3993e5c0ab8af9d44966b271acded35fff9c90b4ac8be7638554: Status 404 returned error can't find the container with id cbcbef4e9f7c3993e5c0ab8af9d44966b271acded35fff9c90b4ac8be7638554
Oct 31 04:45:00.123373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123334    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" event={"ID":"f9e33b65-8e7f-4ce6-8199-8c60671a84e4","Type":"ContainerStarted","Data":"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"}
Oct 31 04:45:00.123798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123396    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" event={"ID":"f9e33b65-8e7f-4ce6-8199-8c60671a84e4","Type":"ContainerStarted","Data":"e419b90b1b7faff4b7b556477be171c14dd583e37b3d0100d4decb55628fb9c1"}
Oct 31 04:45:00.123798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123423    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:45:00.123798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123446    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:45:00.123798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123469    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" event={"ID":"82010fd1-72b3-46f4-9ec8-f9909ade03b8","Type":"ContainerStarted","Data":"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"}
Oct 31 04:45:00.123798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.123492    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" event={"ID":"82010fd1-72b3-46f4-9ec8-f9909ade03b8","Type":"ContainerStarted","Data":"cbcbef4e9f7c3993e5c0ab8af9d44966b271acded35fff9c90b4ac8be7638554"}
Oct 31 04:45:00.164432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.164366    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" podStartSLOduration=2.164349689 podStartE2EDuration="2.164349689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:00.161531812 +0000 UTC m=+2396.194531512" watchObservedRunningTime="2026-10-31 04:45:00.164349689 +0000 UTC m=+2396.197349367"
Oct 31 04:45:00.221637 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.221565    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" podStartSLOduration=2.22154404 podStartE2EDuration="2.22154404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:00.214524177 +0000 UTC m=+2396.247523855" watchObservedRunningTime="2026-10-31 04:45:00.22154404 +0000 UTC m=+2396.254543719"
Oct 31 04:45:00.251204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.249800    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:00.254999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.254967    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.259707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.259676    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:00.264607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.264546    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.283574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.283538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:00.311939 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.311898    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9p7s\" (UniqueName: \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.312849 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.311969    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.312849 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.312007    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkcw\" (UniqueName: \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.312849 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.312076    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.315340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.315307    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:00.321417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.321373    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:00.321640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.321502    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.329774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.329751    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\""
Oct 31 04:45:00.365861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.365770    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:00.369936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.369906    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:00.384510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.384432    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.413897 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.413861    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9v6\" (UniqueName: \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.414257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.414011    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9p7s\" (UniqueName: \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.414257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.414069    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.414257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.414104    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkcw\" (UniqueName: \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.414573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.414480    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.414573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.414526    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.415395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.415371    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.416310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.416286    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.417045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.417025    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:00.444351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.444288    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9p7s\" (UniqueName: \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.444351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.444329    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkcw\" (UniqueName: \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.456195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.455434    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:00.515753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.515723    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9m6\" (UniqueName: \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.515922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.515830    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9v6\" (UniqueName: \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.515966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.515932    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.516064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.516048    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.516665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.516641    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.529607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.529569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:00.540992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.540964    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9v6\" (UniqueName: \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\") pod \"maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.576550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.576500    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:00.584194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.584146    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:00.618475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.617760    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.618475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.617871    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9m6\" (UniqueName: \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.618475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.618427    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.643454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.643294    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9m6\" (UniqueName: \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.681639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.681589    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:00.710342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.710293    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:00.928384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.928304    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:00.932438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:00.932416    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:00.935797 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:45:00.935772    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6405c2_1c25_4d54_a85f_48a74e8ec937.slice/crio-562990333a5ac190dd4097518441501dc0ccaf49619ccfa69d9c54b8eca98806 WatchSource:0}: Error finding container 562990333a5ac190dd4097518441501dc0ccaf49619ccfa69d9c54b8eca98806: Status 404 returned error can't find the container with id 562990333a5ac190dd4097518441501dc0ccaf49619ccfa69d9c54b8eca98806
Oct 31 04:45:00.936491 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:45:00.936474    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaabfa3d_8fee_4a55_9bff_b1dd597cccbc.slice/crio-69b2390ed7505c7b9000d3c27220b89824f5975916b4c86a608e422b6977c967 WatchSource:0}: Error finding container 69b2390ed7505c7b9000d3c27220b89824f5975916b4c86a608e422b6977c967: Status 404 returned error can't find the container with id 69b2390ed7505c7b9000d3c27220b89824f5975916b4c86a608e422b6977c967
Oct 31 04:45:01.020058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.019661    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:01.057736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.057705    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:01.061569 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:45:01.061541    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b28316_4925_484d_92a8_5029f0b4e6c3.slice/crio-114686a7825aa60837372e7d8e0737420cd9ced64fc47d728f65443ce931c7f9 WatchSource:0}: Error finding container 114686a7825aa60837372e7d8e0737420cd9ced64fc47d728f65443ce931c7f9: Status 404 returned error can't find the container with id 114686a7825aa60837372e7d8e0737420cd9ced64fc47d728f65443ce931c7f9
Oct 31 04:45:01.064192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.064113    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:01.125942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.125906    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" event={"ID":"9b6405c2-1c25-4d54-a85f-48a74e8ec937","Type":"ContainerStarted","Data":"562990333a5ac190dd4097518441501dc0ccaf49619ccfa69d9c54b8eca98806"}
Oct 31 04:45:01.127379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.127315    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" event={"ID":"faabfa3d-8fee-4a55-9bff-b1dd597cccbc","Type":"ContainerStarted","Data":"69b2390ed7505c7b9000d3c27220b89824f5975916b4c86a608e422b6977c967"}
Oct 31 04:45:01.129113 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.129084    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" event={"ID":"b7812869-de64-4baa-9130-a7a4d03474c7","Type":"ContainerStarted","Data":"42cceddf951fe06a02f51b753b232f3d10a833210b3baa2db333a7d0a3a6a333"}
Oct 31 04:45:01.130678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.130653    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerStarted","Data":"114686a7825aa60837372e7d8e0737420cd9ced64fc47d728f65443ce931c7f9"}
Oct 31 04:45:01.246240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.243227    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:01.461140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.461096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:01.674571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.674537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:01.891966 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:01.891881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:02.138016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.137980    6931 generic.go:356] "Generic (PLEG): container finished" podID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerID="1558aca0168314fd6fe1565ad378f10f22d08832a62e5a43206a432f6d8fb6ef" exitCode=60
Oct 31 04:45:02.138490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.138062    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" event={"ID":"9b6405c2-1c25-4d54-a85f-48a74e8ec937","Type":"ContainerDied","Data":"1558aca0168314fd6fe1565ad378f10f22d08832a62e5a43206a432f6d8fb6ef"}
Oct 31 04:45:02.138490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.138366    6931 scope.go:122] "RemoveContainer" containerID="1558aca0168314fd6fe1565ad378f10f22d08832a62e5a43206a432f6d8fb6ef"
Oct 31 04:45:02.139805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.139782    6931 generic.go:356] "Generic (PLEG): container finished" podID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerID="a13cb1406b58595f51c76cc525cd57fd276c6b64c3934bdc852dff107362ac5f" exitCode=60
Oct 31 04:45:02.139896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.139826    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" event={"ID":"faabfa3d-8fee-4a55-9bff-b1dd597cccbc","Type":"ContainerDied","Data":"a13cb1406b58595f51c76cc525cd57fd276c6b64c3934bdc852dff107362ac5f"}
Oct 31 04:45:02.140095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.140077    6931 scope.go:122] "RemoveContainer" containerID="a13cb1406b58595f51c76cc525cd57fd276c6b64c3934bdc852dff107362ac5f"
Oct 31 04:45:02.141701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.141548    6931 generic.go:356] "Generic (PLEG): container finished" podID="b7812869-de64-4baa-9130-a7a4d03474c7" containerID="42f713093f8d2529911d1ee7b333214c8e665bb76cbef55c5356cb8218d23419" exitCode=60
Oct 31 04:45:02.141701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.141635    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" event={"ID":"b7812869-de64-4baa-9130-a7a4d03474c7","Type":"ContainerDied","Data":"42f713093f8d2529911d1ee7b333214c8e665bb76cbef55c5356cb8218d23419"}
Oct 31 04:45:02.141826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.141794    6931 scope.go:122] "RemoveContainer" containerID="42f713093f8d2529911d1ee7b333214c8e665bb76cbef55c5356cb8218d23419"
Oct 31 04:45:02.143377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.143356    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerStarted","Data":"976d68afa1efa1ec1a47c351e2b328320424279121478a0c353239628a67c832"}
Oct 31 04:45:02.258425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:02.258371    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" podStartSLOduration=2.258356955 podStartE2EDuration="2.258356955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:02.256398917 +0000 UTC m=+2398.289398596" watchObservedRunningTime="2026-10-31 04:45:02.258356955 +0000 UTC m=+2398.291356635"
Oct 31 04:45:03.151746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.151708    6931 generic.go:356] "Generic (PLEG): container finished" podID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerID="cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e" exitCode=60
Oct 31 04:45:03.152206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.151754    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" event={"ID":"faabfa3d-8fee-4a55-9bff-b1dd597cccbc","Type":"ContainerDied","Data":"cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e"}
Oct 31 04:45:03.152206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.151816    6931 scope.go:122] "RemoveContainer" containerID="a13cb1406b58595f51c76cc525cd57fd276c6b64c3934bdc852dff107362ac5f"
Oct 31 04:45:03.152206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.152089    6931 scope.go:122] "RemoveContainer" containerID="cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e"
Oct 31 04:45:03.152506 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:03.152451    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8_odh-ai-gateway-infra(faabfa3d-8fee-4a55-9bff-b1dd597cccbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc"
Oct 31 04:45:03.153810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.153786    6931 generic.go:356] "Generic (PLEG): container finished" podID="b7812869-de64-4baa-9130-a7a4d03474c7" containerID="6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5" exitCode=60
Oct 31 04:45:03.153901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.153838    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" event={"ID":"b7812869-de64-4baa-9130-a7a4d03474c7","Type":"ContainerDied","Data":"6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5"}
Oct 31 04:45:03.154133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.154113    6931 scope.go:122] "RemoveContainer" containerID="6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5"
Oct 31 04:45:03.154463 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:03.154433    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq_odh-ai-gateway-infra(b7812869-de64-4baa-9130-a7a4d03474c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" podUID="b7812869-de64-4baa-9130-a7a4d03474c7"
Oct 31 04:45:03.155504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.155482    6931 generic.go:356] "Generic (PLEG): container finished" podID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerID="976d68afa1efa1ec1a47c351e2b328320424279121478a0c353239628a67c832" exitCode=60
Oct 31 04:45:03.155623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.155564    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerDied","Data":"976d68afa1efa1ec1a47c351e2b328320424279121478a0c353239628a67c832"}
Oct 31 04:45:03.156036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.156018    6931 scope.go:122] "RemoveContainer" containerID="976d68afa1efa1ec1a47c351e2b328320424279121478a0c353239628a67c832"
Oct 31 04:45:03.159431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.159410    6931 generic.go:356] "Generic (PLEG): container finished" podID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerID="d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949" exitCode=60
Oct 31 04:45:03.159522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.159436    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" event={"ID":"9b6405c2-1c25-4d54-a85f-48a74e8ec937","Type":"ContainerDied","Data":"d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949"}
Oct 31 04:45:03.159778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.159760    6931 scope.go:122] "RemoveContainer" containerID="d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949"
Oct 31 04:45:03.160033 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:03.160011    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf_odh-ai-gateway-infra(9b6405c2-1c25-4d54-a85f-48a74e8ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937"
Oct 31 04:45:03.265793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.265764    6931 scope.go:122] "RemoveContainer" containerID="42f713093f8d2529911d1ee7b333214c8e665bb76cbef55c5356cb8218d23419"
Oct 31 04:45:03.281656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.281635    6931 scope.go:122] "RemoveContainer" containerID="1558aca0168314fd6fe1565ad378f10f22d08832a62e5a43206a432f6d8fb6ef"
Oct 31 04:45:03.300135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:03.300102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:04.147786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.147753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:04.167417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.167386    6931 scope.go:122] "RemoveContainer" containerID="d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949"
Oct 31 04:45:04.167882 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:04.167660    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf_odh-ai-gateway-infra(9b6405c2-1c25-4d54-a85f-48a74e8ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937"
Oct 31 04:45:04.168917 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.168894    6931 scope.go:122] "RemoveContainer" containerID="cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e"
Oct 31 04:45:04.169173 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:04.169131    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8_odh-ai-gateway-infra(faabfa3d-8fee-4a55-9bff-b1dd597cccbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc"
Oct 31 04:45:04.170491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.170474    6931 scope.go:122] "RemoveContainer" containerID="6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5"
Oct 31 04:45:04.170700 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:04.170673    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq_odh-ai-gateway-infra(b7812869-de64-4baa-9130-a7a4d03474c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" podUID="b7812869-de64-4baa-9130-a7a4d03474c7"
Oct 31 04:45:04.172016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.171995    6931 generic.go:356] "Generic (PLEG): container finished" podID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerID="f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1" exitCode=60
Oct 31 04:45:04.172108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.172029    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerDied","Data":"f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1"}
Oct 31 04:45:04.172108 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.172069    6931 scope.go:122] "RemoveContainer" containerID="976d68afa1efa1ec1a47c351e2b328320424279121478a0c353239628a67c832"
Oct 31 04:45:04.172278 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:04.172259    6931 scope.go:122] "RemoveContainer" containerID="f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1"
Oct 31 04:45:04.172478 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:04.172461    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr_odh-ai-gateway-infra(78b28316-4925-484d-92a8-5029f0b4e6c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3"
Oct 31 04:45:05.165004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:05.164957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:05.182263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:05.182185    6931 scope.go:122] "RemoveContainer" containerID="f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1"
Oct 31 04:45:05.182667 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:05.182549    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr_odh-ai-gateway-infra(78b28316-4925-484d-92a8-5029f0b4e6c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3"
Oct 31 04:45:07.038651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:07.038611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:07.253878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:07.253846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:07.462256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:07.462135    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:08.837170 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:08.837123    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:09.099670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:09.099587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:10.226257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:10.226215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:10.435758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:10.435718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:10.779297 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:10.779245    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:45:10.820269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:10.820229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:11.134302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:11.134274    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 04:45:11.134485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:11.134349    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 04:45:12.581941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:12.581905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:14.118786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:14.118757    6931 scope.go:122] "RemoveContainer" containerID="6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5"
Oct 31 04:45:14.119220 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:14.118899    6931 scope.go:122] "RemoveContainer" containerID="d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949"
Oct 31 04:45:14.347855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:14.347826    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:14.932646 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:14.932604    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:15.232773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.232742    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:15.255709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.255671    6931 generic.go:356] "Generic (PLEG): container finished" podID="b7812869-de64-4baa-9130-a7a4d03474c7" containerID="9a02e342d596284c0536c58fb5ee5184a23b23d45d0311c817240770a8afd396" exitCode=60
Oct 31 04:45:15.255873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.255739    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" event={"ID":"b7812869-de64-4baa-9130-a7a4d03474c7","Type":"ContainerDied","Data":"9a02e342d596284c0536c58fb5ee5184a23b23d45d0311c817240770a8afd396"}
Oct 31 04:45:15.255873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.255801    6931 scope.go:122] "RemoveContainer" containerID="6d8a211db92b9debe415b15433959792e4e15c617f1b2dcae04af78eeef794e5"
Oct 31 04:45:15.256254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.256234    6931 scope.go:122] "RemoveContainer" containerID="9a02e342d596284c0536c58fb5ee5184a23b23d45d0311c817240770a8afd396"
Oct 31 04:45:15.256579 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:15.256557    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq_odh-ai-gateway-infra(b7812869-de64-4baa-9130-a7a4d03474c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" podUID="b7812869-de64-4baa-9130-a7a4d03474c7"
Oct 31 04:45:15.258012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.257992    6931 generic.go:356] "Generic (PLEG): container finished" podID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerID="5f2c248c52fa5d1ef6f1736c8f84acf8ab5206f2849aea5f3284a888fbbab5c3" exitCode=60
Oct 31 04:45:15.258097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.258065    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" event={"ID":"9b6405c2-1c25-4d54-a85f-48a74e8ec937","Type":"ContainerDied","Data":"5f2c248c52fa5d1ef6f1736c8f84acf8ab5206f2849aea5f3284a888fbbab5c3"}
Oct 31 04:45:15.258479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.258460    6931 scope.go:122] "RemoveContainer" containerID="5f2c248c52fa5d1ef6f1736c8f84acf8ab5206f2849aea5f3284a888fbbab5c3"
Oct 31 04:45:15.258803 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:15.258785    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf_odh-ai-gateway-infra(9b6405c2-1c25-4d54-a85f-48a74e8ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937"
Oct 31 04:45:15.276074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.276052    6931 scope.go:122] "RemoveContainer" containerID="d0b1085bd109420a93e1df88b297fedad5b09deed4208d6657edf77955998949"
Oct 31 04:45:15.481571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:15.481490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:16.029823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.029783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:16.237410 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.237371    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:16.332695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.332596    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:16.412189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.387379    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:16.459820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.459786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:16.598409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.598384    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:16.600625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.600603    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:16.684087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684054    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\" (UniqueName: \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\") pod \"b7812869-de64-4baa-9130-a7a4d03474c7\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") "
Oct 31 04:45:16.684280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684113    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\" (UniqueName: \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\") pod \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") "
Oct 31 04:45:16.684280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684176    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\") pod \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\" (UID: \"9b6405c2-1c25-4d54-a85f-48a74e8ec937\") "
Oct 31 04:45:16.684414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684338    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\") pod \"b7812869-de64-4baa-9130-a7a4d03474c7\" (UID: \"b7812869-de64-4baa-9130-a7a4d03474c7\") "
Oct 31 04:45:16.684839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684590    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca" pod "9b6405c2-1c25-4d54-a85f-48a74e8ec937" (UID: "9b6405c2-1c25-4d54-a85f-48a74e8ec937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:16.684839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684658    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca" pod "b7812869-de64-4baa-9130-a7a4d03474c7" (UID: "b7812869-de64-4baa-9130-a7a4d03474c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:16.684999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684855    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7812869-de64-4baa-9130-a7a4d03474c7-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:16.684999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.684872    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6405c2-1c25-4d54-a85f-48a74e8ec937-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:16.686896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.686873    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s" pod "9b6405c2-1c25-4d54-a85f-48a74e8ec937" (UID: "9b6405c2-1c25-4d54-a85f-48a74e8ec937"). InnerVolumeSpecName "kube-api-access-n9p7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:16.687003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.686911    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6" pod "b7812869-de64-4baa-9130-a7a4d03474c7" (UID: "b7812869-de64-4baa-9130-a7a4d03474c7"). InnerVolumeSpecName "kube-api-access-xw9m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:16.786592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.786555    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9m6\" (UniqueName: \"kubernetes.io/projected/b7812869-de64-4baa-9130-a7a4d03474c7-kube-api-access-xw9m6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:16.786592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:16.786585    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9p7s\" (UniqueName: \"kubernetes.io/projected/9b6405c2-1c25-4d54-a85f-48a74e8ec937-kube-api-access-n9p7s\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:17.114962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.114929    6931 scope.go:122] "RemoveContainer" containerID="cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e"
Oct 31 04:45:17.115203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.115056    6931 scope.go:122] "RemoveContainer" containerID="f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1"
Oct 31 04:45:17.315866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.315837    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"
Oct 31 04:45:17.316300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.315887    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq" event={"ID":"b7812869-de64-4baa-9130-a7a4d03474c7","Type":"ContainerDied","Data":"42cceddf951fe06a02f51b753b232f3d10a833210b3baa2db333a7d0a3a6a333"}
Oct 31 04:45:17.316300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.315957    6931 scope.go:122] "RemoveContainer" containerID="9a02e342d596284c0536c58fb5ee5184a23b23d45d0311c817240770a8afd396"
Oct 31 04:45:17.317474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.317451    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"
Oct 31 04:45:17.317578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.317457    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf" event={"ID":"9b6405c2-1c25-4d54-a85f-48a74e8ec937","Type":"ContainerDied","Data":"562990333a5ac190dd4097518441501dc0ccaf49619ccfa69d9c54b8eca98806"}
Oct 31 04:45:17.326587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.326564    6931 scope.go:122] "RemoveContainer" containerID="5f2c248c52fa5d1ef6f1736c8f84acf8ab5206f2849aea5f3284a888fbbab5c3"
Oct 31 04:45:17.383858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.383790    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:17.391327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.391300    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6ac82-29890365-ws4tq"]
Oct 31 04:45:17.442273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.442236    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:17.446415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:17.446390    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ded435-29890365-rrplf"]
Oct 31 04:45:18.131500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.131464    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" path="/var/lib/kubelet/pods/9b6405c2-1c25-4d54-a85f-48a74e8ec937/volumes"
Oct 31 04:45:18.131881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.131866    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" path="/var/lib/kubelet/pods/b7812869-de64-4baa-9130-a7a4d03474c7/volumes"
Oct 31 04:45:18.280941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.280901    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:18.325846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.325769    6931 generic.go:356] "Generic (PLEG): container finished" podID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerID="6f360b7b8ac1f332353092169bd6a78eada5f95649aa89f4d628d25a1969aa5c" exitCode=60
Oct 31 04:45:18.326348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.325847    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" event={"ID":"faabfa3d-8fee-4a55-9bff-b1dd597cccbc","Type":"ContainerDied","Data":"6f360b7b8ac1f332353092169bd6a78eada5f95649aa89f4d628d25a1969aa5c"}
Oct 31 04:45:18.326348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.325903    6931 scope.go:122] "RemoveContainer" containerID="cd3b59259c274d64ec7740f7f45d01cff0605cb55772fd032a8739da9349a15e"
Oct 31 04:45:18.326348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.326309    6931 scope.go:122] "RemoveContainer" containerID="6f360b7b8ac1f332353092169bd6a78eada5f95649aa89f4d628d25a1969aa5c"
Oct 31 04:45:18.326657 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:18.326636    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8_odh-ai-gateway-infra(faabfa3d-8fee-4a55-9bff-b1dd597cccbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc"
Oct 31 04:45:18.328687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.328668    6931 generic.go:356] "Generic (PLEG): container finished" podID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerID="5ba42f4c636613b3349b3756fee13ff022f7ade6b0980ef561262a7b6a2a73b7" exitCode=60
Oct 31 04:45:18.328776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.328745    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerDied","Data":"5ba42f4c636613b3349b3756fee13ff022f7ade6b0980ef561262a7b6a2a73b7"}
Oct 31 04:45:18.329068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.329053    6931 scope.go:122] "RemoveContainer" containerID="5ba42f4c636613b3349b3756fee13ff022f7ade6b0980ef561262a7b6a2a73b7"
Oct 31 04:45:18.329319 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:18.329301    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr_odh-ai-gateway-infra(78b28316-4925-484d-92a8-5029f0b4e6c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3"
Oct 31 04:45:18.338000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:18.337983    6931 scope.go:122] "RemoveContainer" containerID="f47ef27482e087fb057070dac8fad8753dc5f8a5844c9af61f7ee23c297edbb1"
Oct 31 04:45:19.377171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.377122    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:19.425821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.425786    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:19.562359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.562321    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:19.624184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.624127    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\" (UniqueName: \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\") pod \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") "
Oct 31 04:45:19.624362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.624260    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\") pod \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\" (UID: \"faabfa3d-8fee-4a55-9bff-b1dd597cccbc\") "
Oct 31 04:45:19.624622 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.624599    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca" pod "faabfa3d-8fee-4a55-9bff-b1dd597cccbc" (UID: "faabfa3d-8fee-4a55-9bff-b1dd597cccbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:19.625190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.625146    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:19.626511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.626400    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw" pod "faabfa3d-8fee-4a55-9bff-b1dd597cccbc" (UID: "faabfa3d-8fee-4a55-9bff-b1dd597cccbc"). InnerVolumeSpecName "kube-api-access-nfkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:19.656211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.656181    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:19.726091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.726060    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\") pod \"78b28316-4925-484d-92a8-5029f0b4e6c3\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") "
Oct 31 04:45:19.726324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.726268    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\" (UniqueName: \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\") pod \"78b28316-4925-484d-92a8-5029f0b4e6c3\" (UID: \"78b28316-4925-484d-92a8-5029f0b4e6c3\") "
Oct 31 04:45:19.726483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.726460    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca" pod "78b28316-4925-484d-92a8-5029f0b4e6c3" (UID: "78b28316-4925-484d-92a8-5029f0b4e6c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:19.726626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.726611    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b28316-4925-484d-92a8-5029f0b4e6c3-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:19.726688 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.726627    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkcw\" (UniqueName: \"kubernetes.io/projected/faabfa3d-8fee-4a55-9bff-b1dd597cccbc-kube-api-access-nfkcw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:19.728421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.728394    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6" pod "78b28316-4925-484d-92a8-5029f0b4e6c3" (UID: "78b28316-4925-484d-92a8-5029f0b4e6c3"). InnerVolumeSpecName "kube-api-access-2m9v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:19.827575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:19.827540    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9v6\" (UniqueName: \"kubernetes.io/projected/78b28316-4925-484d-92a8-5029f0b4e6c3-kube-api-access-2m9v6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:20.169826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.169793    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:20.346485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.346456    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"
Oct 31 04:45:20.346657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.346454    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8" event={"ID":"faabfa3d-8fee-4a55-9bff-b1dd597cccbc","Type":"ContainerDied","Data":"69b2390ed7505c7b9000d3c27220b89824f5975916b4c86a608e422b6977c967"}
Oct 31 04:45:20.346657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.346590    6931 scope.go:122] "RemoveContainer" containerID="6f360b7b8ac1f332353092169bd6a78eada5f95649aa89f4d628d25a1969aa5c"
Oct 31 04:45:20.347900 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.347881    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr" event={"ID":"78b28316-4925-484d-92a8-5029f0b4e6c3","Type":"ContainerDied","Data":"114686a7825aa60837372e7d8e0737420cd9ced64fc47d728f65443ce931c7f9"}
Oct 31 04:45:20.347995 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.347947    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"
Oct 31 04:45:20.356134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.356115    6931 scope.go:122] "RemoveContainer" containerID="5ba42f4c636613b3349b3756fee13ff022f7ade6b0980ef561262a7b6a2a73b7"
Oct 31 04:45:20.397989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.397959    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:20.403820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.403761    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-965f61-29890365-n6srr"]
Oct 31 04:45:20.437704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.437675    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:20.443190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:20.443139    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c4006d-29890365-hrlh8"]
Oct 31 04:45:22.036214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:22.036140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:22.137814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:22.137777    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" path="/var/lib/kubelet/pods/78b28316-4925-484d-92a8-5029f0b4e6c3/volumes"
Oct 31 04:45:22.138233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:22.138219    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" path="/var/lib/kubelet/pods/faabfa3d-8fee-4a55-9bff-b1dd597cccbc/volumes"
Oct 31 04:45:24.139809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:24.139771    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:26.032145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:26.032100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:26.259436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:26.259402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:26.871994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:26.871957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:27.487907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:27.487858    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:27.709689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:27.709654    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:27.926495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:27.926459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:29.892640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:29.892601    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:31.771669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:31.771627    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:33.628514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:33.628480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:33.949227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:33.946729    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:34.159711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:34.159673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:35.406654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:35.406608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:37.363587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:37.363546    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:37.775914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:37.775779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:37.989998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:37.989953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:39.305953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:39.305915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:40.759949 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:40.759901    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:45:41.051631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:41.051596    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:42.307482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:42.307443    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:45:42.307999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:42.307664    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy" containerID="cri-o://b67bcd74b0095c0cf7e176e6fb66769fd5070f72840f3f80d78d6635aa57124b" gracePeriod=30
Oct 31 04:45:42.924859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:42.924767    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:43.053566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:43.053532    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:45:43.053782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:43.053760    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" containerID="cri-o://e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26" gracePeriod=30
Oct 31 04:45:44.685981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.685908    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 04:45:44.686559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686545    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686560    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686570    6931 cpu_manager.go:414] "containerMap: removing container" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686576    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686591    6931 cpu_manager.go:414] "containerMap: removing container" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686597    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.686605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686605    6931 cpu_manager.go:414] "containerMap: removing container" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686610    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686641    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686645    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686652    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686657    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686663    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686667    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686673    6931 cpu_manager.go:414] "containerMap: removing container" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686678    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686685    6931 cpu_manager.go:414] "containerMap: removing container" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686689    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686696    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686700    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686784    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.686792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686791    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686799    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686806    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686815    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686823    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686829    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686835    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686841    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:44.687267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.686846    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6405c2-1c25-4d54-a85f-48a74e8ec937" containerName="cleanup"
Oct 31 04:45:44.690484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.690462    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.721176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.721132    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a04e4e-serving-cert\""
Oct 31 04:45:44.724478 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.724449    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 04:45:44.830303 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.830269    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsf4\" (UniqueName: \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.830495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.830331    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.830539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.830510    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.830576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.830549    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.886151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.886112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:44.933868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.933828    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.934149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.934128    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.934408 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.934387    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsf4\" (UniqueName: \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.934569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.934552    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.935749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.935725    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.940507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.940444    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.948738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.948709    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:44.964171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:44.964122    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsf4\" (UniqueName: \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\") pod \"maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:45.004513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:45.004480    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:45.096070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:45.096029    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:45.309681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:45.309650    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 04:45:45.313099 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:45:45.313070    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acf8b09_b2d2_4a36_a1e0_d438a334ad9a.slice/crio-35fa961a496aeb647f66804ee0d24e8042fbbd113ac734b575941c154eb94321 WatchSource:0}: Error finding container 35fa961a496aeb647f66804ee0d24e8042fbbd113ac734b575941c154eb94321: Status 404 returned error can't find the container with id 35fa961a496aeb647f66804ee0d24e8042fbbd113ac734b575941c154eb94321
Oct 31 04:45:45.531302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:45.531266    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" event={"ID":"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a","Type":"ContainerStarted","Data":"35fa961a496aeb647f66804ee0d24e8042fbbd113ac734b575941c154eb94321"}
Oct 31 04:45:45.914864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:45.914825    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:46.155182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:46.137041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:46.540235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:46.540198    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" event={"ID":"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a","Type":"ContainerStarted","Data":"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"}
Oct 31 04:45:46.540424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:46.540308    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:46.549071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:46.549041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:46.588065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:46.588001    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" podStartSLOduration=2.174912171 podStartE2EDuration="2.587987338s" totalImagesPullingTime="413.075167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:46.586045548 +0000 UTC m=+2442.619045228" watchObservedRunningTime="2026-10-31 04:45:46.587987338 +0000 UTC m=+2442.620987016"
Oct 31 04:45:47.089998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.089959    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:47.515623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.515581    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:45:47.515813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.515651    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:45:47.549062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.549032    6931 generic.go:356] "Generic (PLEG): container finished" podID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerID="b67bcd74b0095c0cf7e176e6fb66769fd5070f72840f3f80d78d6635aa57124b" exitCode=0
Oct 31 04:45:47.549247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.549096    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" event={"ID":"717b6bab-513f-46e5-b202-30c8c71bf5cf","Type":"ContainerDied","Data":"b67bcd74b0095c0cf7e176e6fb66769fd5070f72840f3f80d78d6635aa57124b"}
Oct 31 04:45:47.604575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.604546    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:45:47.681567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681479    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681520    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681591    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681628    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681738    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681795    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681827    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:47.681987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681908    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.681987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.681952    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.682073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682015    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\") pod \"717b6bab-513f-46e5-b202-30c8c71bf5cf\" (UID: \"717b6bab-513f-46e5-b202-30c8c71bf5cf\") "
Oct 31 04:45:47.682073 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682045    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:47.682323 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682303    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:47.682570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682531    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:47.682570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682563    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.682773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682592    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.682773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682610    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/717b6bab-513f-46e5-b202-30c8c71bf5cf-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.682773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.682666    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:47.684547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.684516    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:47.684727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.684709    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "kube-api-access-r58p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:47.685056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.685034    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:45:47.685139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.685075    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy" pod "717b6bab-513f-46e5-b202-30c8c71bf5cf" (UID: "717b6bab-513f-46e5-b202-30c8c71bf5cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:47.784001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.783955    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.784001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.783990    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r58p8\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-kube-api-access-r58p8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.784001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.784007    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.784306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.784019    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/717b6bab-513f-46e5-b202-30c8c71bf5cf-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.784306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.784031    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:47.784306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:47.784042    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/717b6bab-513f-46e5-b202-30c8c71bf5cf-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.314125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.314093    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:45:48.392047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392002    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392096    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392139    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392253    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392325    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392364    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392432    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392500    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392663    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\") pod \"87ba26d8-2892-4a05-b8b9-5ad452474494\" (UID: \"87ba26d8-2892-4a05-b8b9-5ad452474494\") "
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392747    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:48.392970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.392875    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393095    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393130    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393389    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393512    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393537    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393553    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ba26d8-2892-4a05-b8b9-5ad452474494-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.393842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.393567    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.395205 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.395140    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:45:48.395731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.395705    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:45:48.395838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.395787    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:48.396024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.395994    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh" pod "87ba26d8-2892-4a05-b8b9-5ad452474494" (UID: "87ba26d8-2892-4a05-b8b9-5ad452474494"). InnerVolumeSpecName "kube-api-access-gg2nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:45:48.495135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.495100    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.495135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.495132    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.495135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.495142    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ba26d8-2892-4a05-b8b9-5ad452474494-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.495393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.495150    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ba26d8-2892-4a05-b8b9-5ad452474494-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.495393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.495192    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2nh\" (UniqueName: \"kubernetes.io/projected/87ba26d8-2892-4a05-b8b9-5ad452474494-kube-api-access-gg2nh\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:45:48.556570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.556529    6931 generic.go:356] "Generic (PLEG): container finished" podID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerID="e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26" exitCode=0
Oct 31 04:45:48.556730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.556583    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"
Oct 31 04:45:48.556730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.556612    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" event={"ID":"87ba26d8-2892-4a05-b8b9-5ad452474494","Type":"ContainerDied","Data":"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"}
Oct 31 04:45:48.556730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.556694    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts" event={"ID":"87ba26d8-2892-4a05-b8b9-5ad452474494","Type":"ContainerDied","Data":"91e8561fed5ae728ed44c454e3036ea7d4e71f68139ea077f81946a30482d7e8"}
Oct 31 04:45:48.556730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.556721    6931 scope.go:122] "RemoveContainer" containerID="e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"
Oct 31 04:45:48.558438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.558418    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"
Oct 31 04:45:48.558527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.558444    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp" event={"ID":"717b6bab-513f-46e5-b202-30c8c71bf5cf","Type":"ContainerDied","Data":"660cad8e02a6003d5b928bd1a015beb0c7139778beb133cae7f4404db878f5d6"}
Oct 31 04:45:48.566915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.566896    6931 scope.go:122] "RemoveContainer" containerID="e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"
Oct 31 04:45:48.567259 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:45:48.567236    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26\": container with ID starting with e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26 not found: ID does not exist" containerID="e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"
Oct 31 04:45:48.567334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.567269    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26"} err="failed to get container status \"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26\": rpc error: code = NotFound desc = could not find container \"e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26\": container with ID starting with e76beb72c80793b0874ef545aebf58d149b67829b65481408a7ae78b33c34e26 not found: ID does not exist"
Oct 31 04:45:48.567334 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.567287    6931 scope.go:122] "RemoveContainer" containerID="b67bcd74b0095c0cf7e176e6fb66769fd5070f72840f3f80d78d6635aa57124b"
Oct 31 04:45:48.609068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.608992    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:45:48.614654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.614629    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-g57ts"]
Oct 31 04:45:48.648168 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.648124    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:45:48.652082 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.652057    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-s44tp"]
Oct 31 04:45:48.909649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:48.909560    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:49.717512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:49.717462    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:49.939907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:49.939874    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:50.045679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.045639    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:45:50.046281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046266    6931 cpu_manager.go:414] "containerMap: removing container" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy"
Oct 31 04:45:50.046332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046284    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy"
Oct 31 04:45:50.046332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046309    6931 cpu_manager.go:414] "containerMap: removing container" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:50.046332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046314    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b28316-4925-484d-92a8-5029f0b4e6c3" containerName="cleanup"
Oct 31 04:45:50.046332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046322    6931 cpu_manager.go:414] "containerMap: removing container" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy"
Oct 31 04:45:50.046332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046327    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy"
Oct 31 04:45:50.046501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046344    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:50.046501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046356    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:50.046501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046472    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" containerName="istio-proxy"
Oct 31 04:45:50.046501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046482    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="faabfa3d-8fee-4a55-9bff-b1dd597cccbc" containerName="cleanup"
Oct 31 04:45:50.046501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046495    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7812869-de64-4baa-9130-a7a4d03474c7" containerName="cleanup"
Oct 31 04:45:50.046653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.046504    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" containerName="istio-proxy"
Oct 31 04:45:50.051307 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.051291    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.066067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.066046    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"agc-cred-34c4b3c7af7f\""
Oct 31 04:45:50.069610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.069583    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:45:50.078365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.078338    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-e2e--a8f34224-dockercfg-s86rw\""
Oct 31 04:45:50.078615 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.078599    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-567ca6\""
Oct 31 04:45:50.078702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.078610    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"kube-root-ca.crt\""
Oct 31 04:45:50.078702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.078617    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"openshift-service-ca.crt\""
Oct 31 04:45:50.078784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.078709    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-567ca6\"/\"routing-overlay\""
Oct 31 04:45:50.118189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.118134    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld4w\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.118338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.118268    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.118338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.118298    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.118523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.118501    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.123460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.123421    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717b6bab-513f-46e5-b202-30c8c71bf5cf" path="/var/lib/kubelet/pods/717b6bab-513f-46e5-b202-30c8c71bf5cf/volumes"
Oct 31 04:45:50.124013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.123991    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ba26d8-2892-4a05-b8b9-5ad452474494" path="/var/lib/kubelet/pods/87ba26d8-2892-4a05-b8b9-5ad452474494/volumes"
Oct 31 04:45:50.219666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.219624    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.219891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.219742    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld4w\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.219891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.219804    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.219891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.219831    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.221470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.220870    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.221470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.221428    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.222969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.222947    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.280968 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.280677    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld4w\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\") pod \"payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") " pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.365409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.365324    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:50.680522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.680441    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:45:50.683902 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:45:50.683874    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0035e7c9_1f0b_4800_a4db_5209bb1f9158.slice/crio-dd3dd5052dd7bba99a85db178bd70a22e94d2d9dc055a7c44e218b137f9dc9ed WatchSource:0}: Error finding container dd3dd5052dd7bba99a85db178bd70a22e94d2d9dc055a7c44e218b137f9dc9ed: Status 404 returned error can't find the container with id dd3dd5052dd7bba99a85db178bd70a22e94d2d9dc055a7c44e218b137f9dc9ed
Oct 31 04:45:50.908942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:50.908905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:51.582835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:51.582799    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" event={"ID":"0035e7c9-1f0b-4800-a4db-5209bb1f9158","Type":"ContainerStarted","Data":"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"}
Oct 31 04:45:51.583063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:51.582869    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:45:51.583063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:51.582895    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" event={"ID":"0035e7c9-1f0b-4800-a4db-5209bb1f9158","Type":"ContainerStarted","Data":"dd3dd5052dd7bba99a85db178bd70a22e94d2d9dc055a7c44e218b137f9dc9ed"}
Oct 31 04:45:51.626694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:51.626635    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" podStartSLOduration=1.626615234 podStartE2EDuration="1.626615234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:45:51.624308399 +0000 UTC m=+2447.657308102" watchObservedRunningTime="2026-10-31 04:45:51.626615234 +0000 UTC m=+2447.659614918"
Oct 31 04:45:52.554722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:52.554692    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 04:45:52.637922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:52.637878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:52.850275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:52.850190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:54.563263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:54.563226    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:56.343568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:56.343530    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:58.107338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:58.107304    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:59.110945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:59.110908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:59.331616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:59.331578    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:59.619632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:59.619599    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:59.836960 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:45:59.836919    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:00.056524 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:00.056473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:00.462185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:00.462084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:00.838179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:00.837228    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:02.282181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:02.276408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:02.509881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:02.509844    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:02.592366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:02.592272    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:46:02.683944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:02.683906    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:46:02.722521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:02.722243    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-34c4b3c7af7f: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:02.722521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:02.722272    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:02.722521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:02.722334    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:03.222316945 +0000 UTC m=+2459.255316602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:02.722521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:02.722253    6931 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-plugins-e2e-extmod-b-567ca6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:02.722521 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:02.722427    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:03.222405481 +0000 UTC m=+2459.255405157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:02.835189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:02.835143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:03.045988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:03.045948    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:03.229627 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:03.229582    6931 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-plugins-e2e-extmod-b-567ca6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:03.229825 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:03.229686    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:04.229667185 +0000 UTC m=+2460.262666859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:03.229825 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:03.229597    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-34c4b3c7af7f: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:03.229825 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:03.229708    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:03.229825 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:03.229738    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:04.229729891 +0000 UTC m=+2460.262729548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:04.250885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:04.245614    6931 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-plugins-e2e-extmod-b-567ca6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:04.250885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:04.245693    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:06.245673243 +0000 UTC m=+2462.278672904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:04.250885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:04.246343    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-34c4b3c7af7f: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:04.250885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:04.246370    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:04.250885 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:04.246418    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:06.246401291 +0000 UTC m=+2462.279400963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:04.271553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:04.271517    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:06.054460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:06.054407    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:06.270247 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:06.270213    6931 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-plugins-e2e-extmod-b-567ca6: configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:06.270444 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:06.270279    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-567ca6/agc-cred-34c4b3c7af7f: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:06.270444 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:06.270299    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg: secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:06.270444 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:06.270284    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:10.270269467 +0000 UTC m=+2466.303269124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-567ca6" not found
Oct 31 04:46:06.270444 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:06.270358    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials podName:0035e7c9-1f0b-4800-a4db-5209bb1f9158 nodeName:}" failed. No retries permitted until 2026-10-31 04:46:10.270337291 +0000 UTC m=+2466.303336953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158") : secret "agc-cred-34c4b3c7af7f" not found
Oct 31 04:46:07.685094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:07.685048    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" podUID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" containerName="payload-processing" containerID="cri-o://bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd" gracePeriod=25
Oct 31 04:46:07.935012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:07.934911    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:08.017919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.017886    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:46:08.093724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.093690    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\") pod \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") "
Oct 31 04:46:08.093934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.093776    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\") pod \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") "
Oct 31 04:46:08.093934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.093917    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\") pod \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") "
Oct 31 04:46:08.094106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.094088    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\") pod \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\" (UID: \"0035e7c9-1f0b-4800-a4db-5209bb1f9158\") "
Oct 31 04:46:08.094668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.094407    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume" pod "0035e7c9-1f0b-4800-a4db-5209bb1f9158" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:46:08.094668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.094621    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume" pod "0035e7c9-1f0b-4800-a4db-5209bb1f9158" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:46:08.094668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.094640    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:46:08.096078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.096057    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials" pod "0035e7c9-1f0b-4800-a4db-5209bb1f9158" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:46:08.096470 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.096443    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w" pod "0035e7c9-1f0b-4800-a4db-5209bb1f9158" (UID: "0035e7c9-1f0b-4800-a4db-5209bb1f9158"). InnerVolumeSpecName "kube-api-access-wld4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:46:08.196922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.196645    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wld4w\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-kube-api-access-wld4w\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:46:08.196922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.196690    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0035e7c9-1f0b-4800-a4db-5209bb1f9158-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:46:08.196922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.196706    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0035e7c9-1f0b-4800-a4db-5209bb1f9158-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:46:08.707183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.707107    6931 generic.go:356] "Generic (PLEG): container finished" podID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" containerID="bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd" exitCode=0
Oct 31 04:46:08.707678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.707373    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" event={"ID":"0035e7c9-1f0b-4800-a4db-5209bb1f9158","Type":"ContainerDied","Data":"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"}
Oct 31 04:46:08.707678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.707481    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg" event={"ID":"0035e7c9-1f0b-4800-a4db-5209bb1f9158","Type":"ContainerDied","Data":"dd3dd5052dd7bba99a85db178bd70a22e94d2d9dc055a7c44e218b137f9dc9ed"}
Oct 31 04:46:08.707678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.707524    6931 scope.go:122] "RemoveContainer" containerID="bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"
Oct 31 04:46:08.707992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.707973    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"
Oct 31 04:46:08.722953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.722930    6931 scope.go:122] "RemoveContainer" containerID="bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"
Oct 31 04:46:08.723306 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:08.723280    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd\": container with ID starting with bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd not found: ID does not exist" containerID="bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"
Oct 31 04:46:08.723411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.723313    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd"} err="failed to get container status \"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd\": rpc error: code = NotFound desc = could not find container \"bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd\": container with ID starting with bfbac0ea83aeab3d763a2b27160448f0a57174cb059423f098eba972ead855cd not found: ID does not exist"
Oct 31 04:46:08.760044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.760013    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:46:08.771925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:08.771891    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-567ca6/payload-processing-external-model-e2e-extmod-b-567ca6-67f8cl8zg"]
Oct 31 04:46:09.168050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:09.168014    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:09.388671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:09.388638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:09.709045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:09.709009    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:10.123386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:10.123354    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" path="/var/lib/kubelet/pods/0035e7c9-1f0b-4800-a4db-5209bb1f9158/volumes"
Oct 31 04:46:10.758951 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:10.758918    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:46:11.463397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:11.463362    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:13.305754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:13.305707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:15.233218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:15.233178    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:15.467468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:15.467435    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:15.699508 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:15.699467    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:15.918723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:15.918678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:16.129327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:16.129284    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:16.711336 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:16.711302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:16.941331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:16.941286    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:17.491590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:17.491548    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:18.291505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:18.291462    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:19.241412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:19.241376    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:19.455889 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:19.455845    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:19.672311 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:19.672273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:19.896093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:19.896058    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:20.533859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.533828    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:46:20.534475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.534459    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" containerName="payload-processing"
Oct 31 04:46:20.534526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.534476    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" containerName="payload-processing"
Oct 31 04:46:20.534585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.534576    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0035e7c9-1f0b-4800-a4db-5209bb1f9158" containerName="payload-processing"
Oct 31 04:46:20.539132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.539113    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.570513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570487    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"routing-overlay\""
Oct 31 04:46:20.570513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570491    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-plugins-e2e-extmod-a-401761\""
Oct 31 04:46:20.570744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570563    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"openshift-service-ca.crt\""
Oct 31 04:46:20.570744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570596    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-e2e--201094b0-dockercfg-fx78q\""
Oct 31 04:46:20.570744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570606    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"kube-root-ca.crt\""
Oct 31 04:46:20.570744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.570671    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"e2e-cfl-adeb95-provider-cred\""
Oct 31 04:46:20.574871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.574844    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:46:20.657051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.657002    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.657281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.657216    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.657281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.657254    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.657397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.657291    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52s5j\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.758663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.758625    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.758863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.758675    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.758863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.758717    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52s5j\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.758863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.758806    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.759437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.759412    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.759536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.759506    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.761166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.761139    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.781874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.781851    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52s5j\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\") pod \"payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:20.852961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:20.852874    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:21.156951 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.156873    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:46:21.160826 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:46:21.160797    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a34771_37a0_4744_8830_944fee9c2df4.slice/crio-f9f7cdacb3bc8061deea50dc9b3018ae7943290647222cca8484a46387f7ce19 WatchSource:0}: Error finding container f9f7cdacb3bc8061deea50dc9b3018ae7943290647222cca8484a46387f7ce19: Status 404 returned error can't find the container with id f9f7cdacb3bc8061deea50dc9b3018ae7943290647222cca8484a46387f7ce19
Oct 31 04:46:21.492943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.492843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:21.803291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.803252    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" event={"ID":"27a34771-37a0-4744-8830-944fee9c2df4","Type":"ContainerStarted","Data":"b8afd1db5b9c3a80658fb50687d2ace0cc4ef6b6663fab6658d3621e20705ac8"}
Oct 31 04:46:21.803725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.803310    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:21.803725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.803327    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" event={"ID":"27a34771-37a0-4744-8830-944fee9c2df4","Type":"ContainerStarted","Data":"f9f7cdacb3bc8061deea50dc9b3018ae7943290647222cca8484a46387f7ce19"}
Oct 31 04:46:21.843724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:21.843650    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" podStartSLOduration=1.843631816 podStartE2EDuration="1.843631816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:46:21.843297811 +0000 UTC m=+2477.876297495" watchObservedRunningTime="2026-10-31 04:46:21.843631816 +0000 UTC m=+2477.876631496"
Oct 31 04:46:23.344497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:23.344459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:25.204980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:25.204939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:27.015231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:27.015194    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:28.793072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:28.793031    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:30.519958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:30.519923    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:32.304432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:32.304398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:32.812562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:32.812533    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:46:34.097147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:34.097113    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:35.852869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:35.852829    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:37.731826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:37.731793    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:39.026821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:39.026787    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:39.525071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:39.525023    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:40.228776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:40.228740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:40.486681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:40.486580    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:40.694166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:40.694132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:40.759586 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:46:40.759500    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:46:40.913049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:40.913015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:41.155524 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:41.155484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:41.440690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:41.440598    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:41.661875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:41.661839    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:41.879231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:41.879193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:43.417871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:43.417832    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:43.683392 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:43.683304    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:44.806825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:44.806784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:45.279036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:45.278945    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:47.012256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:47.012215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:48.816387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:48.816347    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:50.619969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:50.619930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:52.399744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:52.399709    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:54.195580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:54.195542    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:56.092185 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:56.092118    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:57.906447 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:57.906407    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:58.335740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:58.335700    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:46:58.335961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:58.335935    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" containerID="cri-o://d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e" gracePeriod=30
Oct 31 04:46:59.686075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:59.686036    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:59.990927 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:46:59.990841    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:00.579704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:00.579670    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:01.418621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:01.418589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:02.580132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:02.580094    6931 patch_prober.go:29] interesting pod/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:47:02.580538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:02.580183    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:47:03.312305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.312265    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:03.609485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.609458    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:47:03.755291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755256    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755302    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755357    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755422    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755483    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755515    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755573    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755560    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:47:03.755855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755663    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.755855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755646    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:47:03.755855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755728    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:47:03.755855 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755739    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\") pod \"6ba4a620-55af-4497-9549-a21577316dec\" (UID: \"6ba4a620-55af-4497-9549-a21577316dec\") "
Oct 31 04:47:03.756044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755931    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:47:03.756044 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.755951    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:47:03.756270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.756248    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4a620-55af-4497-9549-a21577316dec-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.756270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.756268    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.756424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.756277    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.756424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.756284    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.756424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.756297    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.757732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.757710    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:47:03.757830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.757815    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:47:03.758227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.758207    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "kube-api-access-z9mk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:47:03.758227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.758219    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token" pod "6ba4a620-55af-4497-9549-a21577316dec" (UID: "6ba4a620-55af-4497-9549-a21577316dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:47:03.857651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.857561    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.857651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.857594    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba4a620-55af-4497-9549-a21577316dec-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.857651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.857605    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mk2\" (UniqueName: \"kubernetes.io/projected/6ba4a620-55af-4497-9549-a21577316dec-kube-api-access-z9mk2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:03.857651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:03.857618    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba4a620-55af-4497-9549-a21577316dec-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:04.107935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.107846    6931 generic.go:356] "Generic (PLEG): container finished" podID="6ba4a620-55af-4497-9549-a21577316dec" containerID="d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e" exitCode=0
Oct 31 04:47:04.107935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.107907    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"
Oct 31 04:47:04.108179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.107930    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" event={"ID":"6ba4a620-55af-4497-9549-a21577316dec","Type":"ContainerDied","Data":"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"}
Oct 31 04:47:04.108179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.108013    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd" event={"ID":"6ba4a620-55af-4497-9549-a21577316dec","Type":"ContainerDied","Data":"7debc97eaa4241949b422d773e347294fc8912e58ecea3380958b416e6d11962"}
Oct 31 04:47:04.108179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.108036    6931 scope.go:122] "RemoveContainer" containerID="d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"
Oct 31 04:47:04.118234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.118216    6931 scope.go:122] "RemoveContainer" containerID="d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"
Oct 31 04:47:04.118496 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:47:04.118476    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e\": container with ID starting with d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e not found: ID does not exist" containerID="d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"
Oct 31 04:47:04.118565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.118506    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e"} err="failed to get container status \"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e\": rpc error: code = NotFound desc = could not find container \"d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e\": container with ID starting with d03bf31e058677dc87177e7bce199ef1b190a0f999cff0f0e738740794a6da9e not found: ID does not exist"
Oct 31 04:47:04.174652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.174621    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:47:04.186196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:04.186169    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4c2387-openshift-default-5b68d44445-2msbd"]
Oct 31 04:47:05.036546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.036498    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:05.250942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.250859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:05.607249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.607213    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:47:05.607796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.607784    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy"
Oct 31 04:47:05.607841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.607798    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy"
Oct 31 04:47:05.607925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.607916    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba4a620-55af-4497-9549-a21577316dec" containerName="istio-proxy"
Oct 31 04:47:05.612551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.612534    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.629463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.629424    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"openshift-service-ca.crt\""
Oct 31 04:47:05.641293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.641245    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"routing-overlay\""
Oct 31 04:47:05.641293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.641266    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"agc-cred-7194275e28ab\""
Oct 31 04:47:05.641293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.641288    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"kube-root-ca.crt\""
Oct 31 04:47:05.641561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.641257    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a99f2a\""
Oct 31 04:47:05.641561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.641408    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a99f2a\"/\"payload-processing-external-model-e2e--c07b99ee-dockercfg-gzb75\""
Oct 31 04:47:05.644099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.644075    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:47:05.790212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.790123    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.790476 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.790378    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjt9s\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.790558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.790495    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.790611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.790557    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.891977    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.892039    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.892117    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.892252    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt9s\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892758 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.892732    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.892876 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.892820    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.894587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.894568    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.915801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.915773    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt9s\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\") pod \"payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") " pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:05.926667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:05.926638    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:06.123084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:06.123047    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba4a620-55af-4497-9549-a21577316dec" path="/var/lib/kubelet/pods/6ba4a620-55af-4497-9549-a21577316dec/volumes"
Oct 31 04:47:06.229929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:06.229856    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:47:06.233652 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:47:06.233626    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609b9a57_b918_4b00_adab_64c8d4a77307.slice/crio-13a376955340d7333b937ac20faf5d62f213db5c26dd2ef11dc4ae4352f21b15 WatchSource:0}: Error finding container 13a376955340d7333b937ac20faf5d62f213db5c26dd2ef11dc4ae4352f21b15: Status 404 returned error can't find the container with id 13a376955340d7333b937ac20faf5d62f213db5c26dd2ef11dc4ae4352f21b15
Oct 31 04:47:06.655293 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:06.655258    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:07.029883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:07.029849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:07.143892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:07.143853    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" event={"ID":"609b9a57-b918-4b00-adab-64c8d4a77307","Type":"ContainerStarted","Data":"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"}
Oct 31 04:47:07.144363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:07.143912    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" event={"ID":"609b9a57-b918-4b00-adab-64c8d4a77307","Type":"ContainerStarted","Data":"13a376955340d7333b937ac20faf5d62f213db5c26dd2ef11dc4ae4352f21b15"}
Oct 31 04:47:07.144363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:07.143938    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:07.185595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:07.185531    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" podStartSLOduration=2.185515105 podStartE2EDuration="2.185515105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:07.183757081 +0000 UTC m=+2523.216756760" watchObservedRunningTime="2026-10-31 04:47:07.185515105 +0000 UTC m=+2523.218514785"
Oct 31 04:47:08.841550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:08.841515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:10.697610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:10.697577    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:10.759800 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:47:10.759761    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:47:12.329583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:12.329543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:12.655016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:12.654926    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:14.268829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:14.268740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:15.987797 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:15.987750    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:17.709708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:17.709664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:18.153152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:18.153120    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:47:19.492731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:19.492693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:21.235450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:21.235415    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:23.185396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:23.185351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:25.111233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:25.111193    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:26.386824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:26.386780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:26.593177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:26.593091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:27.277129 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.277077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:27.709138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.709106    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:47:27.713775 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.713752    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.740885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740851    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-fcdfce9e22b3\""
Oct 31 04:47:27.740885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740882    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\""
Oct 31 04:47:27.741142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740851    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-cc70fa128d1a\""
Oct 31 04:47:27.741142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740889    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\""
Oct 31 04:47:27.741142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740922    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-72313fc13ae2\""
Oct 31 04:47:27.741142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.740989    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\""
Oct 31 04:47:27.741526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.741502    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-055b438a3215\""
Oct 31 04:47:27.741654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.741639    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-f7kv4\""
Oct 31 04:47:27.741705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.741687    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\""
Oct 31 04:47:27.744262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.744243    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:47:27.788418 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.788380    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.788614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.788438    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.788614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.788543    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.788710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.788659    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd9f\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.889987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.889946    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.890196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.890065    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd9f\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.890255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.890229    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.890308 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.890278    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.891060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.891030    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.891186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.891118    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.892863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.892833    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:27.915896 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:27.915864    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd9f\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:28.029384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:28.029299    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:28.336147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:28.336086    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:47:28.339382 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:47:28.339344    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52604e08_fb05_4fed_a9d8_7e66a7567359.slice/crio-f3cad1e1fc9f82ac129951b8ef81b8bd76e30f1ad3b1c875db39612ff0b45ff3 WatchSource:0}: Error finding container f3cad1e1fc9f82ac129951b8ef81b8bd76e30f1ad3b1c875db39612ff0b45ff3: Status 404 returned error can't find the container with id f3cad1e1fc9f82ac129951b8ef81b8bd76e30f1ad3b1c875db39612ff0b45ff3
Oct 31 04:47:29.155498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:29.155458    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:29.285238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:29.285200    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" event={"ID":"52604e08-fb05-4fed-a9d8-7e66a7567359","Type":"ContainerStarted","Data":"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"}
Oct 31 04:47:29.285413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:29.285298    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" event={"ID":"52604e08-fb05-4fed-a9d8-7e66a7567359","Type":"ContainerStarted","Data":"f3cad1e1fc9f82ac129951b8ef81b8bd76e30f1ad3b1c875db39612ff0b45ff3"}
Oct 31 04:47:29.285413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:29.285322    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:29.327360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:29.327302    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" podStartSLOduration=2.327283382 podStartE2EDuration="2.327283382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:29.325146038 +0000 UTC m=+2545.358145739" watchObservedRunningTime="2026-10-31 04:47:29.327283382 +0000 UTC m=+2545.360283063"
Oct 31 04:47:31.007272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:31.007233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:32.907458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:32.907421    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:34.801769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:34.801735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:36.675189 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:36.675129    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:37.204708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:37.204673    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:47:37.354228 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:37.354191    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:37.590818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:37.590775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:37.922415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:37.922257    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:38.169604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:38.169570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:38.745083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:38.745036    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:38.974803 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:38.974764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:39.190535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:39.190502    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:39.671633 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:39.671595    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:39.886402 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:39.886355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:40.294887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:40.294859    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:47:40.606462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:40.606337    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:40.759050 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:47:40.759013    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:47:42.205435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.205395    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" podUID="27a34771-37a0-4744-8830-944fee9c2df4" containerName="payload-processing" containerID="cri-o://b8afd1db5b9c3a80658fb50687d2ace0cc4ef6b6663fab6658d3621e20705ac8" gracePeriod=25
Oct 31 04:47:42.349519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.349477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:42.370267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.370232    6931 generic.go:356] "Generic (PLEG): container finished" podID="27a34771-37a0-4744-8830-944fee9c2df4" containerID="b8afd1db5b9c3a80658fb50687d2ace0cc4ef6b6663fab6658d3621e20705ac8" exitCode=0
Oct 31 04:47:42.370431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.370302    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" event={"ID":"27a34771-37a0-4744-8830-944fee9c2df4","Type":"ContainerDied","Data":"b8afd1db5b9c3a80658fb50687d2ace0cc4ef6b6663fab6658d3621e20705ac8"}
Oct 31 04:47:42.489450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.489421    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:47:42.577023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.576992    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\") pod \"27a34771-37a0-4744-8830-944fee9c2df4\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") "
Oct 31 04:47:42.577207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577177    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\") pod \"27a34771-37a0-4744-8830-944fee9c2df4\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") "
Oct 31 04:47:42.577249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577221    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\") pod \"27a34771-37a0-4744-8830-944fee9c2df4\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") "
Oct 31 04:47:42.577338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577319    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\") pod \"27a34771-37a0-4744-8830-944fee9c2df4\" (UID: \"27a34771-37a0-4744-8830-944fee9c2df4\") "
Oct 31 04:47:42.577601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577579    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume" pod "27a34771-37a0-4744-8830-944fee9c2df4" (UID: "27a34771-37a0-4744-8830-944fee9c2df4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:47:42.577673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577613    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume" pod "27a34771-37a0-4744-8830-944fee9c2df4" (UID: "27a34771-37a0-4744-8830-944fee9c2df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:47:42.577770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577756    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:42.577823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.577773    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a34771-37a0-4744-8830-944fee9c2df4-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:42.579822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.579796    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j" pod "27a34771-37a0-4744-8830-944fee9c2df4" (UID: "27a34771-37a0-4744-8830-944fee9c2df4"). InnerVolumeSpecName "kube-api-access-52s5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:47:42.579915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.579818    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials" pod "27a34771-37a0-4744-8830-944fee9c2df4" (UID: "27a34771-37a0-4744-8830-944fee9c2df4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:47:42.678961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.678920    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:42.678961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:42.678955    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52s5j\" (UniqueName: \"kubernetes.io/projected/27a34771-37a0-4744-8830-944fee9c2df4-kube-api-access-52s5j\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:47:43.378351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:43.378314    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm" event={"ID":"27a34771-37a0-4744-8830-944fee9c2df4","Type":"ContainerDied","Data":"f9f7cdacb3bc8061deea50dc9b3018ae7943290647222cca8484a46387f7ce19"}
Oct 31 04:47:43.378351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:43.378333    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"
Oct 31 04:47:43.378875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:43.378379    6931 scope.go:122] "RemoveContainer" containerID="b8afd1db5b9c3a80658fb50687d2ace0cc4ef6b6663fab6658d3621e20705ac8"
Oct 31 04:47:43.428759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:43.428725    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:47:43.440357 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:43.440316    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-558bqsdsm"]
Oct 31 04:47:44.123099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:44.123065    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a34771-37a0-4744-8830-944fee9c2df4" path="/var/lib/kubelet/pods/27a34771-37a0-4744-8830-944fee9c2df4/volumes"
Oct 31 04:47:44.126777 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:44.126744    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:45.932832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:45.932794    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:46.221330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:46.221241    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:46.441942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:46.441885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:47.899514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:47.899470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:48.106077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:48.106027    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:48.350057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:48.350021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:48.806541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:48.806503    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:49.047292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:49.047253    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:49.272977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:49.272888    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:49.524911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:49.524825    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:49.814587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:49.814539    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:50.417732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:50.417690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:50.639177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:50.638984    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:51.022150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:51.022114    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:51.232570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:51.232534    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:51.949472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:51.949439    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:52.380502 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:52.380466    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:53.027266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.027229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:53.240034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.239994    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:53.293328 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.293292    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:47:53.293893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.293880    6931 cpu_manager.go:414] "containerMap: removing container" podUID="27a34771-37a0-4744-8830-944fee9c2df4" containerName="payload-processing"
Oct 31 04:47:53.293943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.293894    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a34771-37a0-4744-8830-944fee9c2df4" containerName="payload-processing"
Oct 31 04:47:53.294030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.294020    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a34771-37a0-4744-8830-944fee9c2df4" containerName="payload-processing"
Oct 31 04:47:53.298929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.298909    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.314642 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.314598    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a04e4e\""
Oct 31 04:47:53.314975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.314916    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"openshift-service-ca.crt\""
Oct 31 04:47:53.326404 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.326370    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"routing-overlay\""
Oct 31 04:47:53.326625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.326422    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-59f7f64fabf6\""
Oct 31 04:47:53.326625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.326421    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-7cae9f4970bf\""
Oct 31 04:47:53.326768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.326514    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"kube-root-ca.crt\""
Oct 31 04:47:53.326768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.326580    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-e2e--ae39f93f-dockercfg-ncvq5\""
Oct 31 04:47:53.329361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.329336    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:47:53.413141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.413108    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.413348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.413150    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7c9\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.413348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.413237    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.413348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.413259    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.514672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.514638    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.514856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.514683    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7c9\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.514856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.514802    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.514856 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.514835    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.515597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.515542    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.515722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.515688    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.517170 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.517138    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.537247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.537222    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7c9\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.619723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.619636    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:53.908458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.908341    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:53.936139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:53.936103    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:47:53.940971 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:47:53.940941    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6155bbc_3a2b_4961_acb1_63d4fd4ed324.slice/crio-d55a4a1ea72962457a2dffa7a75f8bbce6a1eafbb1d3d893c8357bb109802b4f WatchSource:0}: Error finding container d55a4a1ea72962457a2dffa7a75f8bbce6a1eafbb1d3d893c8357bb109802b4f: Status 404 returned error can't find the container with id d55a4a1ea72962457a2dffa7a75f8bbce6a1eafbb1d3d893c8357bb109802b4f
Oct 31 04:47:54.123152 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.123119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:54.356140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.356097    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:54.463990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.463952    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" event={"ID":"c6155bbc-3a2b-4961-acb1-63d4fd4ed324","Type":"ContainerStarted","Data":"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"}
Oct 31 04:47:54.464192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.464018    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" event={"ID":"c6155bbc-3a2b-4961-acb1-63d4fd4ed324","Type":"ContainerStarted","Data":"d55a4a1ea72962457a2dffa7a75f8bbce6a1eafbb1d3d893c8357bb109802b4f"}
Oct 31 04:47:54.464247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.464201    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:47:54.503852 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:54.503787    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" podStartSLOduration=1.5037688299999998 podStartE2EDuration="1.50376883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:54.503123589 +0000 UTC m=+2570.536123270" watchObservedRunningTime="2026-10-31 04:47:54.50376883 +0000 UTC m=+2570.536768513"
Oct 31 04:47:55.474659 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:55.474623    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:56.168609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:56.168570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:57.644624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:57.644587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:57.856373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:57.856333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:58.068178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:58.065091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:59.009177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:58.994189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:59.205667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:59.205629    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:59.756811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:59.756774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:59.889471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:47:59.889434    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:48:00.074561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:00.074489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:00.304207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:00.302810    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:00.774913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:00.774881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:00.979977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:00.979936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:02.126738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:02.126695    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:48:02.127439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:02.127418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:02.129769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:02.129751    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:48:03.909217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:03.909182    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:04.890926 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:04.890879    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" podUID="609b9a57-b918-4b00-adab-64c8d4a77307" containerName="payload-processing" containerID="cri-o://b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449" gracePeriod=25
Oct 31 04:48:05.155957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.155892    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:48:05.294916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.294885    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\") pod \"609b9a57-b918-4b00-adab-64c8d4a77307\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") "
Oct 31 04:48:05.295090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.294973    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\") pod \"609b9a57-b918-4b00-adab-64c8d4a77307\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") "
Oct 31 04:48:05.295090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295030    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\") pod \"609b9a57-b918-4b00-adab-64c8d4a77307\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") "
Oct 31 04:48:05.295188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295137    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\") pod \"609b9a57-b918-4b00-adab-64c8d4a77307\" (UID: \"609b9a57-b918-4b00-adab-64c8d4a77307\") "
Oct 31 04:48:05.295485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295461    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume" pod "609b9a57-b918-4b00-adab-64c8d4a77307" (UID: "609b9a57-b918-4b00-adab-64c8d4a77307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:05.295587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295509    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume" pod "609b9a57-b918-4b00-adab-64c8d4a77307" (UID: "609b9a57-b918-4b00-adab-64c8d4a77307"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:05.295651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295637    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:05.295705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.295651    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609b9a57-b918-4b00-adab-64c8d4a77307-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:05.297190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.297152    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials" pod "609b9a57-b918-4b00-adab-64c8d4a77307" (UID: "609b9a57-b918-4b00-adab-64c8d4a77307"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:05.297399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.297381    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s" pod "609b9a57-b918-4b00-adab-64c8d4a77307" (UID: "609b9a57-b918-4b00-adab-64c8d4a77307"). InnerVolumeSpecName "kube-api-access-hjt9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:05.396699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.396662    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:05.396699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.396691    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjt9s\" (UniqueName: \"kubernetes.io/projected/609b9a57-b918-4b00-adab-64c8d4a77307-kube-api-access-hjt9s\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:05.541743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.541709    6931 generic.go:356] "Generic (PLEG): container finished" podID="609b9a57-b918-4b00-adab-64c8d4a77307" containerID="b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449" exitCode=0
Oct 31 04:48:05.541909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.541766    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"
Oct 31 04:48:05.541909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.541788    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" event={"ID":"609b9a57-b918-4b00-adab-64c8d4a77307","Type":"ContainerDied","Data":"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"}
Oct 31 04:48:05.541909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.541869    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz" event={"ID":"609b9a57-b918-4b00-adab-64c8d4a77307","Type":"ContainerDied","Data":"13a376955340d7333b937ac20faf5d62f213db5c26dd2ef11dc4ae4352f21b15"}
Oct 31 04:48:05.541909 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.541893    6931 scope.go:122] "RemoveContainer" containerID="b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"
Oct 31 04:48:05.553613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.553587    6931 scope.go:122] "RemoveContainer" containerID="b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"
Oct 31 04:48:05.553933 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:05.553910    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449\": container with ID starting with b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449 not found: ID does not exist" containerID="b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"
Oct 31 04:48:05.554047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.553943    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449"} err="failed to get container status \"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449\": rpc error: code = NotFound desc = could not find container \"b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449\": container with ID starting with b17b381909e9ee95e3e655af1a69009e72ec17aba12b09d8daf0c83620e35449 not found: ID does not exist"
Oct 31 04:48:05.596782 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.596742    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:48:05.600245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.600217    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a99f2a/payload-processing-external-model-e2e-extmod-b-a99f2a-7bc558qvz"]
Oct 31 04:48:05.618198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.618166    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:05.843952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:05.843864    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:06.060638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:06.060601    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:06.124134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:06.124056    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609b9a57-b918-4b00-adab-64c8d4a77307" path="/var/lib/kubelet/pods/609b9a57-b918-4b00-adab-64c8d4a77307/volumes"
Oct 31 04:48:07.063771 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.063703    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:07.127292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.127228    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" podUID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" containerName="payload-processing" containerID="cri-o://2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de" gracePeriod=25
Oct 31 04:48:07.275267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.275187    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:07.397548 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.397514    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:48:07.522801 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.522767    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\") pod \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") "
Oct 31 04:48:07.522985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.522826    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\") pod \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") "
Oct 31 04:48:07.522985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.522852    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\") pod \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") "
Oct 31 04:48:07.523122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.523046    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\") pod \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\" (UID: \"c6155bbc-3a2b-4961-acb1-63d4fd4ed324\") "
Oct 31 04:48:07.523621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.523280    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume" pod "c6155bbc-3a2b-4961-acb1-63d4fd4ed324" (UID: "c6155bbc-3a2b-4961-acb1-63d4fd4ed324"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:07.523621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.523431    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume" pod "c6155bbc-3a2b-4961-acb1-63d4fd4ed324" (UID: "c6155bbc-3a2b-4961-acb1-63d4fd4ed324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:07.523621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.523539    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:07.523621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.523559    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:07.525442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.525410    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9" pod "c6155bbc-3a2b-4961-acb1-63d4fd4ed324" (UID: "c6155bbc-3a2b-4961-acb1-63d4fd4ed324"). InnerVolumeSpecName "kube-api-access-hj7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:07.525546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.525447    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials" pod "c6155bbc-3a2b-4961-acb1-63d4fd4ed324" (UID: "c6155bbc-3a2b-4961-acb1-63d4fd4ed324"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:07.557691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.557658    6931 generic.go:356] "Generic (PLEG): container finished" podID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" containerID="2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de" exitCode=0
Oct 31 04:48:07.557841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.557704    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"
Oct 31 04:48:07.557841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.557747    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" event={"ID":"c6155bbc-3a2b-4961-acb1-63d4fd4ed324","Type":"ContainerDied","Data":"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"}
Oct 31 04:48:07.557841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.557807    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj" event={"ID":"c6155bbc-3a2b-4961-acb1-63d4fd4ed324","Type":"ContainerDied","Data":"d55a4a1ea72962457a2dffa7a75f8bbce6a1eafbb1d3d893c8357bb109802b4f"}
Oct 31 04:48:07.557841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.557832    6931 scope.go:122] "RemoveContainer" containerID="2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"
Oct 31 04:48:07.567344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.567328    6931 scope.go:122] "RemoveContainer" containerID="2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"
Oct 31 04:48:07.567587 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:07.567567    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de\": container with ID starting with 2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de not found: ID does not exist" containerID="2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"
Oct 31 04:48:07.567638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.567594    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de"} err="failed to get container status \"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de\": rpc error: code = NotFound desc = could not find container \"2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de\": container with ID starting with 2157e516d538a417822d0fde32e8963dd897f293720ea7028649286b600c24de not found: ID does not exist"
Oct 31 04:48:07.612566 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.611024    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:48:07.641332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.632357    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7c9\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-kube-api-access-hj7c9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:07.641332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.632394    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6155bbc-3a2b-4961-acb1-63d4fd4ed324-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:07.641332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.637251    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-947dqlxhj"]
Oct 31 04:48:07.684187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:07.683140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:08.123421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:08.123386    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" path="/var/lib/kubelet/pods/c6155bbc-3a2b-4961-acb1-63d4fd4ed324/volumes"
Oct 31 04:48:09.718217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:09.718175    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:10.760009 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:10.759962    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:48:11.545924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:11.545876    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:12.106706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:12.106665    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:12.343090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:12.343040    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:12.572534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:12.572489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:12.800020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:12.799980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:13.021724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:13.021638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:13.241982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:13.241948    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:13.456219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:13.456183    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:13.685755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:13.685724    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:14.039096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:14.039047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:14.376231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:14.376133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:15.257658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.257608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:15.469596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.469557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:15.575136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575098    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:15.575759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575744    6931 cpu_manager.go:414] "containerMap: removing container" podUID="609b9a57-b918-4b00-adab-64c8d4a77307" containerName="payload-processing"
Oct 31 04:48:15.575804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575761    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609b9a57-b918-4b00-adab-64c8d4a77307" containerName="payload-processing"
Oct 31 04:48:15.575804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575801    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" containerName="payload-processing"
Oct 31 04:48:15.575877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575807    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" containerName="payload-processing"
Oct 31 04:48:15.575911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575894    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6155bbc-3a2b-4961-acb1-63d4fd4ed324" containerName="payload-processing"
Oct 31 04:48:15.575911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.575907    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="609b9a57-b918-4b00-adab-64c8d4a77307" containerName="payload-processing"
Oct 31 04:48:15.580791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.580775    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.599386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.599354    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:15.608057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.608022    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"agc-cred-603fbee8e306\""
Oct 31 04:48:15.608262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.608073    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"payload-processing-external-model-plugins-e2e-extmod-a-afc519\""
Oct 31 04:48:15.608262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.608096    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"kube-root-ca.crt\""
Oct 31 04:48:15.608262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.608104    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"routing-overlay\""
Oct 31 04:48:15.608533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.608504    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"openshift-service-ca.crt\""
Oct 31 04:48:15.610967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.610943    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"payload-processing-external-model-e2e--49628f74-dockercfg-xg7x2\""
Oct 31 04:48:15.654218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.654182    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.654396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.654327    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.654396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.654386    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.654575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.654559    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7m9p\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.756118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.756079    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7m9p\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.756332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.756132    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.756332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.756265    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.756332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.756321    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.756894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.756873    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.757029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.757008    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.758842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.758820    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.779208 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.779143    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7m9p\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:15.895021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:15.894938    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:16.122487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.122406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:16.201568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.201480    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:16.205818 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:48:16.205779    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3034d7ed_6c3b_4fb9_8a79_eb782ad59c07.slice/crio-f6b7b9acc925fc6c3ce11ce77488eda99bbeb69af991bbc8935946b96484e656 WatchSource:0}: Error finding container f6b7b9acc925fc6c3ce11ce77488eda99bbeb69af991bbc8935946b96484e656: Status 404 returned error can't find the container with id f6b7b9acc925fc6c3ce11ce77488eda99bbeb69af991bbc8935946b96484e656
Oct 31 04:48:16.285665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.285633    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:48:16.289806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.289783    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.308518 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.308478    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-afc519\"/\"agc-cred-7f7ae7f543df\""
Oct 31 04:48:16.308978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.308955    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:48:16.362785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.362733    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxk2\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.362785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.362787    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.363045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.363019    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.363107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.363078    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.464551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.464457    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.464551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.464506    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.464795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.464572    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxk2\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.464795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.464593    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.465277 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.465255    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.465378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.465333    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.467072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.467053    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.487458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.487434    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxk2\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\") pod \"payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") " pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.607935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.607893    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:16.629934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.629899    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" event={"ID":"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07","Type":"ContainerStarted","Data":"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"}
Oct 31 04:48:16.630090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.629956    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" event={"ID":"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07","Type":"ContainerStarted","Data":"f6b7b9acc925fc6c3ce11ce77488eda99bbeb69af991bbc8935946b96484e656"}
Oct 31 04:48:16.630090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.629997    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:16.674731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.674666    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" podStartSLOduration=1.6746414569999999 podStartE2EDuration="1.674641457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:16.672240237 +0000 UTC m=+2592.705239917" watchObservedRunningTime="2026-10-31 04:48:16.674641457 +0000 UTC m=+2592.707641139"
Oct 31 04:48:16.914424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:16.914388    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:48:16.917899 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:48:16.917866    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46225329_5550_46d2_9bfb_7b9218f34640.slice/crio-08d86f14441eb4370e9fdab762e8ba608df3e3445d04ffd9a72ac2735c504118 WatchSource:0}: Error finding container 08d86f14441eb4370e9fdab762e8ba608df3e3445d04ffd9a72ac2735c504118: Status 404 returned error can't find the container with id 08d86f14441eb4370e9fdab762e8ba608df3e3445d04ffd9a72ac2735c504118
Oct 31 04:48:17.636963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:17.636922    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" event={"ID":"46225329-5550-46d2-9bfb-7b9218f34640","Type":"ContainerStarted","Data":"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"}
Oct 31 04:48:17.637473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:17.636979    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" event={"ID":"46225329-5550-46d2-9bfb-7b9218f34640","Type":"ContainerStarted","Data":"08d86f14441eb4370e9fdab762e8ba608df3e3445d04ffd9a72ac2735c504118"}
Oct 31 04:48:17.637473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:17.637077    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:17.683805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:17.683743    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" podStartSLOduration=1.6837168089999999 podStartE2EDuration="1.683716809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:17.682295399 +0000 UTC m=+2593.715295097" watchObservedRunningTime="2026-10-31 04:48:17.683716809 +0000 UTC m=+2593.716716489"
Oct 31 04:48:18.238083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:18.238039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:19.732074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:19.732010    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:19.944607 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:19.944567    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:20.049761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:20.049728    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:48:20.235466 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:20.235431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:22.097032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:22.096991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:24.221371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:24.221324    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:25.050473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.050421    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" podUID="52604e08-fb05-4fed-a9d8-7e66a7567359" containerName="payload-processing" containerID="cri-o://7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d" gracePeriod=25
Oct 31 04:48:25.352831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.352803    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:48:25.382497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382463    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\") pod \"52604e08-fb05-4fed-a9d8-7e66a7567359\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") "
Oct 31 04:48:25.382656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382570    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\") pod \"52604e08-fb05-4fed-a9d8-7e66a7567359\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") "
Oct 31 04:48:25.382656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382646    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\") pod \"52604e08-fb05-4fed-a9d8-7e66a7567359\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") "
Oct 31 04:48:25.382764 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382722    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\") pod \"52604e08-fb05-4fed-a9d8-7e66a7567359\" (UID: \"52604e08-fb05-4fed-a9d8-7e66a7567359\") "
Oct 31 04:48:25.383054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382953    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume" pod "52604e08-fb05-4fed-a9d8-7e66a7567359" (UID: "52604e08-fb05-4fed-a9d8-7e66a7567359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:25.383054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.382973    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume" pod "52604e08-fb05-4fed-a9d8-7e66a7567359" (UID: "52604e08-fb05-4fed-a9d8-7e66a7567359"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:25.383371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.383346    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:25.383538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.383514    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52604e08-fb05-4fed-a9d8-7e66a7567359-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:25.385517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.385489    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials" pod "52604e08-fb05-4fed-a9d8-7e66a7567359" (UID: "52604e08-fb05-4fed-a9d8-7e66a7567359"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:25.385631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.385544    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f" pod "52604e08-fb05-4fed-a9d8-7e66a7567359" (UID: "52604e08-fb05-4fed-a9d8-7e66a7567359"). InnerVolumeSpecName "kube-api-access-pdd9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:25.484691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.484655    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdd9f\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-kube-api-access-pdd9f\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:25.484691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.484685    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52604e08-fb05-4fed-a9d8-7e66a7567359-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:25.691723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.691627    6931 generic.go:356] "Generic (PLEG): container finished" podID="52604e08-fb05-4fed-a9d8-7e66a7567359" containerID="7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d" exitCode=0
Oct 31 04:48:25.691723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.691688    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"
Oct 31 04:48:25.691915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.691713    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" event={"ID":"52604e08-fb05-4fed-a9d8-7e66a7567359","Type":"ContainerDied","Data":"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"}
Oct 31 04:48:25.691915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.691817    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4" event={"ID":"52604e08-fb05-4fed-a9d8-7e66a7567359","Type":"ContainerDied","Data":"f3cad1e1fc9f82ac129951b8ef81b8bd76e30f1ad3b1c875db39612ff0b45ff3"}
Oct 31 04:48:25.691915 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.691853    6931 scope.go:122] "RemoveContainer" containerID="7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"
Oct 31 04:48:25.702184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.702126    6931 scope.go:122] "RemoveContainer" containerID="7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"
Oct 31 04:48:25.702491 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:25.702465    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d\": container with ID starting with 7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d not found: ID does not exist" containerID="7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"
Oct 31 04:48:25.702549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.702501    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d"} err="failed to get container status \"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d\": rpc error: code = NotFound desc = could not find container \"7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d\": container with ID starting with 7888d45e013d2fc9288937a2bd3c092a9afca0515f1db37fd4164c6b09603c3d not found: ID does not exist"
Oct 31 04:48:25.750388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.750344    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:48:25.762050 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:25.762020    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6c74jjhz4"]
Oct 31 04:48:26.124723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:26.124689    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52604e08-fb05-4fed-a9d8-7e66a7567359" path="/var/lib/kubelet/pods/52604e08-fb05-4fed-a9d8-7e66a7567359/volumes"
Oct 31 04:48:26.247722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:26.247688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:26.963816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:26.963783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:27.176547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:27.176515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:27.452436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:27.452395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:27.640226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:27.640194    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:27.680197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:27.679342    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:28.621813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:28.621776    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:28.646890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:28.646861    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:48:28.719321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:28.719288    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:28.839177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:28.839124    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:29.988846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:29.988810    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:30.211487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.211445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:30.693611 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.693573    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:30.811417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.811385    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:30.812031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.812018    6931 cpu_manager.go:414] "containerMap: removing container" podUID="52604e08-fb05-4fed-a9d8-7e66a7567359" containerName="payload-processing"
Oct 31 04:48:30.812083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.812033    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52604e08-fb05-4fed-a9d8-7e66a7567359" containerName="payload-processing"
Oct 31 04:48:30.812203 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.812193    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="52604e08-fb05-4fed-a9d8-7e66a7567359" containerName="payload-processing"
Oct 31 04:48:30.825242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.823719    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.838031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.838002    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:30.844820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.844788    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-055b438a3215\""
Oct 31 04:48:30.845062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.844835    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-cc70fa128d1a\""
Oct 31 04:48:30.845062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.844841    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-f7kv4\""
Oct 31 04:48:30.845289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.844906    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\""
Oct 31 04:48:30.853398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.853373    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\""
Oct 31 04:48:30.853601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.853373    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\""
Oct 31 04:48:30.853601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.853400    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\""
Oct 31 04:48:30.879088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.879051    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7lq\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.879088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.879093    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.879348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.879294    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.879443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.879426    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.980818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.980717    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7lq\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.980993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.980851    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.980993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.980955    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.981118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.981083    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.981887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.981857    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.982236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.982181    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:30.983910 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:30.983890    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:31.006302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.006273    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7lq\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:31.145572 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.145542    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:31.447335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.447136    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:31.450498 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:48:31.450469    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba51d733_f9c5_4ec2_baf4_a10612a643a1.slice/crio-9f4cd2e142719b9c80ac93779dc8723afff06af1c536c752638a4c7d391327af WatchSource:0}: Error finding container 9f4cd2e142719b9c80ac93779dc8723afff06af1c536c752638a4c7d391327af: Status 404 returned error can't find the container with id 9f4cd2e142719b9c80ac93779dc8723afff06af1c536c752638a4c7d391327af
Oct 31 04:48:31.764496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.764409    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" event={"ID":"ba51d733-f9c5-4ec2-baf4-a10612a643a1","Type":"ContainerStarted","Data":"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"}
Oct 31 04:48:31.764496 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.764462    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" event={"ID":"ba51d733-f9c5-4ec2-baf4-a10612a643a1","Type":"ContainerStarted","Data":"9f4cd2e142719b9c80ac93779dc8723afff06af1c536c752638a4c7d391327af"}
Oct 31 04:48:31.764694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.764585    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:31.805930 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:31.805868    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" podStartSLOduration=1.8058521810000001 podStartE2EDuration="1.805852181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:31.804262671 +0000 UTC m=+2607.837262375" watchObservedRunningTime="2026-10-31 04:48:31.805852181 +0000 UTC m=+2607.838851866"
Oct 31 04:48:32.178644 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:32.178604    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:32.395742 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:32.395698    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:32.746583 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:32.746545    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:32.905991 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:32.905961    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-055b438a3215: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:32.905991 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:32.905991    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:32.906289 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:32.906043    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials podName:ba51d733-f9c5-4ec2-baf4-a10612a643a1 nodeName:}" failed. No retries permitted until 2026-10-31 04:48:33.406025937 +0000 UTC m=+2609.439025593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1") : secret "agc-cred-055b438a3215" not found
Oct 31 04:48:33.341833 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:33.341786    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:33.414127 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:33.414096    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-055b438a3215: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:33.414127 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:33.414128    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:33.414417 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:33.414236    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials podName:ba51d733-f9c5-4ec2-baf4-a10612a643a1 nodeName:}" failed. No retries permitted until 2026-10-31 04:48:34.414216736 +0000 UTC m=+2610.447216393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1") : secret "agc-cred-055b438a3215" not found
Oct 31 04:48:33.719847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:33.719749    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" podUID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" containerName="payload-processing" containerID="cri-o://c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32" gracePeriod=25
Oct 31 04:48:33.988103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:33.988040    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:34.123019 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.122991    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\") pod \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") "
Oct 31 04:48:34.123206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123184    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\") pod \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") "
Oct 31 04:48:34.123281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123215    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\") pod \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") "
Oct 31 04:48:34.123281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123246    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\") pod \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\" (UID: \"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07\") "
Oct 31 04:48:34.123386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123325    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume" pod "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" (UID: "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:34.123631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123605    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume" pod "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" (UID: "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:34.123766 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.123669    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:34.125530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.125504    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p" pod "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" (UID: "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07"). InnerVolumeSpecName "kube-api-access-s7m9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:34.125614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.125574    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials" pod "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" (UID: "3034d7ed-6c3b-4fb9-8a79-eb782ad59c07"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:34.225039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.225006    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:34.225039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.225032    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:34.225039 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.225044    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7m9p\" (UniqueName: \"kubernetes.io/projected/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07-kube-api-access-s7m9p\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:34.427915 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:34.427881    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-055b438a3215: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:34.427915 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:34.427917    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:34.428449 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:34.427978    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials podName:ba51d733-f9c5-4ec2-baf4-a10612a643a1 nodeName:}" failed. No retries permitted until 2026-10-31 04:48:36.427960918 +0000 UTC m=+2612.460960575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1") : secret "agc-cred-055b438a3215" not found
Oct 31 04:48:34.788584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.788554    6931 generic.go:356] "Generic (PLEG): container finished" podID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" containerID="c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32" exitCode=0
Oct 31 04:48:34.788761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.788601    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"
Oct 31 04:48:34.788761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.788637    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" event={"ID":"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07","Type":"ContainerDied","Data":"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"}
Oct 31 04:48:34.788761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.788695    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx" event={"ID":"3034d7ed-6c3b-4fb9-8a79-eb782ad59c07","Type":"ContainerDied","Data":"f6b7b9acc925fc6c3ce11ce77488eda99bbeb69af991bbc8935946b96484e656"}
Oct 31 04:48:34.788761 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.788719    6931 scope.go:122] "RemoveContainer" containerID="c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"
Oct 31 04:48:34.800112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.800095    6931 scope.go:122] "RemoveContainer" containerID="c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"
Oct 31 04:48:34.800420 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:34.800400    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32\": container with ID starting with c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32 not found: ID does not exist" containerID="c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"
Oct 31 04:48:34.800515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.800428    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32"} err="failed to get container status \"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32\": rpc error: code = NotFound desc = could not find container \"c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32\": container with ID starting with c1357966fc7b99d76b12c1a9bbad8a18b9c6bd0c38ff981bac7e99e677ec1b32 not found: ID does not exist"
Oct 31 04:48:34.816365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.816335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:34.840280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.840251    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:34.847847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:34.847817    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-6c9cq75cx"]
Oct 31 04:48:36.123089 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:36.123058    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" path="/var/lib/kubelet/pods/3034d7ed-6c3b-4fb9-8a79-eb782ad59c07/volumes"
Oct 31 04:48:36.454744 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:36.454665    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8973e2/agc-cred-055b438a3215: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:36.454744 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:36.454695    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz: secret "agc-cred-055b438a3215" not found
Oct 31 04:48:36.454933 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:36.454749    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials podName:ba51d733-f9c5-4ec2-baf4-a10612a643a1 nodeName:}" failed. No retries permitted until 2026-10-31 04:48:40.454733795 +0000 UTC m=+2616.487733456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1") : secret "agc-cred-055b438a3215" not found
Oct 31 04:48:36.673687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:36.673649    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:38.416824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:38.416792    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:38.780817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:38.780720    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" podUID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" containerName="payload-processing" containerID="cri-o://c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa" gracePeriod=25
Oct 31 04:48:39.050680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.050619    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:39.190020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.189978    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\") pod \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") "
Oct 31 04:48:39.190244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.190109    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\") pod \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") "
Oct 31 04:48:39.190313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.190277    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\") pod \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") "
Oct 31 04:48:39.190399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.190365    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\") pod \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\" (UID: \"ba51d733-f9c5-4ec2-baf4-a10612a643a1\") "
Oct 31 04:48:39.190651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.190620    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume" pod "ba51d733-f9c5-4ec2-baf4-a10612a643a1" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:39.190724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.190685    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume" pod "ba51d733-f9c5-4ec2-baf4-a10612a643a1" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:39.191076 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.191055    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:39.191175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.191083    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba51d733-f9c5-4ec2-baf4-a10612a643a1-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:39.192380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.192358    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials" pod "ba51d733-f9c5-4ec2-baf4-a10612a643a1" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:39.192589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.192566    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq" pod "ba51d733-f9c5-4ec2-baf4-a10612a643a1" (UID: "ba51d733-f9c5-4ec2-baf4-a10612a643a1"). InnerVolumeSpecName "kube-api-access-dh7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:39.292175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.292132    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:39.292349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.292203    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7lq\" (UniqueName: \"kubernetes.io/projected/ba51d733-f9c5-4ec2-baf4-a10612a643a1-kube-api-access-dh7lq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:39.823967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.823929    6931 generic.go:356] "Generic (PLEG): container finished" podID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" containerID="c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa" exitCode=0
Oct 31 04:48:39.824479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.823985    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"
Oct 31 04:48:39.824479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.824015    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" event={"ID":"ba51d733-f9c5-4ec2-baf4-a10612a643a1","Type":"ContainerDied","Data":"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"}
Oct 31 04:48:39.824479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.824069    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz" event={"ID":"ba51d733-f9c5-4ec2-baf4-a10612a643a1","Type":"ContainerDied","Data":"9f4cd2e142719b9c80ac93779dc8723afff06af1c536c752638a4c7d391327af"}
Oct 31 04:48:39.824479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.824093    6931 scope.go:122] "RemoveContainer" containerID="c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"
Oct 31 04:48:39.834101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.834078    6931 scope.go:122] "RemoveContainer" containerID="c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"
Oct 31 04:48:39.834436 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:39.834415    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa\": container with ID starting with c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa not found: ID does not exist" containerID="c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"
Oct 31 04:48:39.834510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.834444    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa"} err="failed to get container status \"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa\": rpc error: code = NotFound desc = could not find container \"c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa\": container with ID starting with c165e022c495510cb52309c7d8d271afc8042d63d169e88133f72a541580d8aa not found: ID does not exist"
Oct 31 04:48:39.877974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.876697    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:39.888134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:39.888097    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-7bbbx7njz"]
Oct 31 04:48:40.123313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:40.123236    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" path="/var/lib/kubelet/pods/ba51d733-f9c5-4ec2-baf4-a10612a643a1/volumes"
Oct 31 04:48:40.275612 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:40.275581    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:40.759610 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:48:40.759572    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:48:42.017987 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:42.017949    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:43.446973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:43.446938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:43.674192 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:43.674131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:43.948559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:43.948518    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:44.168246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.162329    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:44.796074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796033    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 04:48:44.796729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796712    6931 cpu_manager.go:414] "containerMap: removing container" podUID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" containerName="payload-processing"
Oct 31 04:48:44.796799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796730    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" containerName="payload-processing"
Oct 31 04:48:44.796799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796769    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" containerName="payload-processing"
Oct 31 04:48:44.796799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796774    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" containerName="payload-processing"
Oct 31 04:48:44.796916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796874    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3034d7ed-6c3b-4fb9-8a79-eb782ad59c07" containerName="payload-processing"
Oct 31 04:48:44.796916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.796889    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba51d733-f9c5-4ec2-baf4-a10612a643a1" containerName="payload-processing"
Oct 31 04:48:44.801815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.801794    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.823030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.822992    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 04:48:44.964369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964336    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964413    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964513    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964557    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964628    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964672    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964700    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjzn\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964773    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:44.964821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:44.964800    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.065723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065649    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.065891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065720    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.065891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065758    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.065891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065805    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.065891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065846    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065897    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.065935    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjzn\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066056    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066112    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066299    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066336    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066447    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066534    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.066670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.066574    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.068174 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.068135    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.068563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.068545    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.088811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.088771    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.088940 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.088838    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjzn\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-m56vf\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.117597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.117562    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:45.525232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.521955    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 04:48:45.546619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.531367    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:48:45.546619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.531455    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:48:45.546619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.531493    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:48:45.591643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.591369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:45.807753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.807717    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:45.871077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.871038    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" event={"ID":"e717b2a3-78b9-451e-b35a-26987837c05a","Type":"ContainerStarted","Data":"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"}
Oct 31 04:48:45.871261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.871150    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" event={"ID":"e717b2a3-78b9-451e-b35a-26987837c05a","Type":"ContainerStarted","Data":"a6015e1aae8865f096b53b3e0f33092ffaca45f94f07e1d147a15425f90aa6a9"}
Oct 31 04:48:45.918139 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:45.918064    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podStartSLOduration=1.918046107 podStartE2EDuration="1.918046107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:45.917314251 +0000 UTC m=+2621.950313956" watchObservedRunningTime="2026-10-31 04:48:45.918046107 +0000 UTC m=+2621.951045796"
Oct 31 04:48:46.058878 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:46.058783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:46.122627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:46.122593    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:46.122820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:46.122687    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:46.126414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:46.126389    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:46.286178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:46.283093    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:47.118506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.118471    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:47.118922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.118520    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:47.151177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.150708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:47.296825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.296794    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:47.301177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.301138    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.318819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.318777    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:48:47.318819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.318777    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-11e01939582a\""
Oct 31 04:48:47.353517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.353482    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n4glb\""
Oct 31 04:48:47.357297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.356436    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:48:47.357297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.356521    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:48:47.357297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.356689    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:48:47.359176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.359140    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:47.418703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.418611    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6wl\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.418703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.418688    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.418946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.418784    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.418946 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.418815    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.527202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.523248    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6wl\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.527202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.523351    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.527202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.523413    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.527202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.523445    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.527202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.524667    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.542083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.542049    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.584217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.579078    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.584217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.582783    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6wl\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\") pod \"payload-processing-external-model-98755745d-4s8cg\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") " pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.616501 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.616462    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:47.966004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:47.965908    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:48.118029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.117994    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:48.118228 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.118050    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:48.289091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.289058    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:48.892417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.892376    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" event={"ID":"2b50e4a1-450c-45d3-886c-038d0b07dcd6","Type":"ContainerStarted","Data":"d4777d644e47ed0768de2dda2cf587cdd1087369ca87326f264008f8b8da99c3"}
Oct 31 04:48:48.892613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.892431    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" event={"ID":"2b50e4a1-450c-45d3-886c-038d0b07dcd6","Type":"ContainerStarted","Data":"4f712da190df230ab9c26bc098fd6a516ac47c0d06a81c12cd0e97b58d194558"}
Oct 31 04:48:48.892613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.892462    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:48.962662 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:48.962600    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" podStartSLOduration=1.962585166 podStartE2EDuration="1.962585166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:48:48.961390408 +0000 UTC m=+2624.994390087" watchObservedRunningTime="2026-10-31 04:48:48.962585166 +0000 UTC m=+2624.995584845"
Oct 31 04:48:49.118669 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:49.118634    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:49.118815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:49.118688    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:50.118536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:50.118500    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:50.118955 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:50.118556    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:50.248180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:50.247115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:51.118339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:51.118309    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:51.118614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:51.118380    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:51.771773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:51.771397    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:51.881621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:51.881584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:52.111180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:52.106798    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:52.126180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:52.119559    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:52.126180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:52.119622    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:52.413231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:52.404737    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:52.863330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:52.863293    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:53.092838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:53.092803    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:53.118268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:53.118190    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-m56vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 31 04:48:53.118268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:53.118245    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 31 04:48:54.292586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:54.292552    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:54.293230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:54.293204    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:54.294025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:54.294006    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 04:48:54.305540 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:54.305507    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:56.158333 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:56.158294    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:56.772670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:56.772631    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" podUID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" containerName="payload-processing" containerID="cri-o://d4777d644e47ed0768de2dda2cf587cdd1087369ca87326f264008f8b8da99c3" gracePeriod=25
Oct 31 04:48:56.950989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:56.950891    6931 generic.go:356] "Generic (PLEG): container finished" podID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" containerID="d4777d644e47ed0768de2dda2cf587cdd1087369ca87326f264008f8b8da99c3" exitCode=0
Oct 31 04:48:56.951207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:56.951090    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" event={"ID":"2b50e4a1-450c-45d3-886c-038d0b07dcd6","Type":"ContainerDied","Data":"d4777d644e47ed0768de2dda2cf587cdd1087369ca87326f264008f8b8da99c3"}
Oct 31 04:48:57.048603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.048567    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:57.093292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.093252    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\") pod \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") "
Oct 31 04:48:57.093467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.093420    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\") pod \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") "
Oct 31 04:48:57.093562 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.093547    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\") pod \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") "
Oct 31 04:48:57.093623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.093609    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\") pod \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\" (UID: \"2b50e4a1-450c-45d3-886c-038d0b07dcd6\") "
Oct 31 04:48:57.093695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.093668    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume" pod "2b50e4a1-450c-45d3-886c-038d0b07dcd6" (UID: "2b50e4a1-450c-45d3-886c-038d0b07dcd6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:57.097198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.094125    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:57.097198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.094167    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume" pod "2b50e4a1-450c-45d3-886c-038d0b07dcd6" (UID: "2b50e4a1-450c-45d3-886c-038d0b07dcd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:57.102816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.102773    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials" pod "2b50e4a1-450c-45d3-886c-038d0b07dcd6" (UID: "2b50e4a1-450c-45d3-886c-038d0b07dcd6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:57.102991 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.102835    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl" pod "2b50e4a1-450c-45d3-886c-038d0b07dcd6" (UID: "2b50e4a1-450c-45d3-886c-038d0b07dcd6"). InnerVolumeSpecName "kube-api-access-lq6wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:57.195146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.195110    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b50e4a1-450c-45d3-886c-038d0b07dcd6-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:57.195146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.195147    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6wl\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-kube-api-access-lq6wl\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:57.195703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.195194    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b50e4a1-450c-45d3-886c-038d0b07dcd6-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:48:57.959745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.959708    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg"
Oct 31 04:48:57.959745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.959736    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-98755745d-4s8cg" event={"ID":"2b50e4a1-450c-45d3-886c-038d0b07dcd6","Type":"ContainerDied","Data":"4f712da190df230ab9c26bc098fd6a516ac47c0d06a81c12cd0e97b58d194558"}
Oct 31 04:48:57.960021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:57.959803    6931 scope.go:122] "RemoveContainer" containerID="d4777d644e47ed0768de2dda2cf587cdd1087369ca87326f264008f8b8da99c3"
Oct 31 04:48:58.012731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.012690    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:58.017117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.017080    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:58.024556 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.024524    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-98755745d-4s8cg"]
Oct 31 04:48:58.125186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.125121    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" path="/var/lib/kubelet/pods/2b50e4a1-450c-45d3-886c-038d0b07dcd6/volumes"
Oct 31 04:48:58.247648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.247562    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:58.925000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:58.924947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:59.617872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.617828    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:59.847839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.847808    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 04:48:59.848534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.848518    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" containerName="payload-processing"
Oct 31 04:48:59.848591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.848536    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" containerName="payload-processing"
Oct 31 04:48:59.848632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.848624    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b50e4a1-450c-45d3-886c-038d0b07dcd6" containerName="payload-processing"
Oct 31 04:48:59.853645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.853620    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.857924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.857891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:59.876683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.876520    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 04:48:59.935974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.935936    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.935997    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936067    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936220    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4hm\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936282    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936312    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936417    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936481    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:48:59.936544 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:48:59.936529    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.050547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.050502    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.055225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.055189    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.055860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.053051    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056012    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056091    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056182    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056227    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056347 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056310    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4hm\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.056482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056370    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.059920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.057473    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.059920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.057732    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.059920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.057974    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.059920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.059138    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.065115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.065089    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.067235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.067213    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.067380 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.056396    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.081851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.081821    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.082514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.082492    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4hm\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j7hwx\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.097818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.097785    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:00.169890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.169788    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:00.314479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.314439    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:00.495882 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.495712    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 04:49:00.499301 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:49:00.499273    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ad350c_6196_420e_9d8b_2210b5ff7152.slice/crio-2a492d88e7eba8ac07209bcc17033ec3b770d34a4c53ebb498a6e1a6849865b6 WatchSource:0}: Error finding container 2a492d88e7eba8ac07209bcc17033ec3b770d34a4c53ebb498a6e1a6849865b6: Status 404 returned error can't find the container with id 2a492d88e7eba8ac07209bcc17033ec3b770d34a4c53ebb498a6e1a6849865b6
Oct 31 04:49:00.502029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.501988    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:49:00.502123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.502062    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:49:00.502123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.502102    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 04:49:00.528743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.528713    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:00.915091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.915047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:00.988879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.988841    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" event={"ID":"d8ad350c-6196-420e-9d8b-2210b5ff7152","Type":"ContainerStarted","Data":"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"}
Oct 31 04:49:00.989081 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:00.988908    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" event={"ID":"d8ad350c-6196-420e-9d8b-2210b5ff7152","Type":"ContainerStarted","Data":"2a492d88e7eba8ac07209bcc17033ec3b770d34a4c53ebb498a6e1a6849865b6"}
Oct 31 04:49:01.138183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:01.134879    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:01.172415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:01.171231    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:01.172415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:01.172221    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:01.172415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:01.172272    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:02.170286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.170251    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:02.170703 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.170308    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:02.397534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.396739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:02.956825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.956764    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podStartSLOduration=3.956748862 podStartE2EDuration="3.956748862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:01.040503889 +0000 UTC m=+2637.073503571" watchObservedRunningTime="2026-10-31 04:49:02.956748862 +0000 UTC m=+2638.989748542"
Oct 31 04:49:02.957729 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.957706    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:49:02.962452 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.962041    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:02.980916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.980888    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:49:02.989495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989463    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-45c8a527100b\""
Oct 31 04:49:02.989676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989537    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\""
Oct 31 04:49:02.989676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989564    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-dzlvk\""
Oct 31 04:49:02.989676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989670    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\""
Oct 31 04:49:02.989851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989711    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\""
Oct 31 04:49:02.989851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:02.989754    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\""
Oct 31 04:49:03.122219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.121883    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.122219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.122065    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.122219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.122147    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.122542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.122274    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pzk\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.185745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.185709    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:03.186206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.185777    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:03.223665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.223573    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pzk\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.223822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.223725    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.223864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.223843    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.223905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.223887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.224791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.224761    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.225468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.225444    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.244180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.240826    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.258960 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.258928    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pzk\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.277379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.277349    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:03.611754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.611718    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:49:03.615308 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:49:03.615278    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4879ed6b_8a5f_4b15_ad5c_f3b959eea887.slice/crio-cfb906e65e7fb574ccb274e66c54ef136628711e70153e50c075929ea8b34d10 WatchSource:0}: Error finding container cfb906e65e7fb574ccb274e66c54ef136628711e70153e50c075929ea8b34d10: Status 404 returned error can't find the container with id cfb906e65e7fb574ccb274e66c54ef136628711e70153e50c075929ea8b34d10
Oct 31 04:49:03.752950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:03.752916    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:04.013244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.013118    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" event={"ID":"4879ed6b-8a5f-4b15-ad5c-f3b959eea887","Type":"ContainerStarted","Data":"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"}
Oct 31 04:49:04.013244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.013191    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" event={"ID":"4879ed6b-8a5f-4b15-ad5c-f3b959eea887","Type":"ContainerStarted","Data":"cfb906e65e7fb574ccb274e66c54ef136628711e70153e50c075929ea8b34d10"}
Oct 31 04:49:04.013244 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.013231    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:04.072599 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.072543    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" podStartSLOduration=2.072527049 podStartE2EDuration="2.072527049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:04.072223058 +0000 UTC m=+2640.105222750" watchObservedRunningTime="2026-10-31 04:49:04.072527049 +0000 UTC m=+2640.105526728"
Oct 31 04:49:04.170579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.170536    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:04.170747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.170612    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:04.407129 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.407091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:04.615604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.615564    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:04.822672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:04.822634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:05.118708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:05.118676    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:05.170416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:05.170380    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:05.170600 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:05.170440    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:06.170702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:06.170670    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:06.171196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:06.171141    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:06.353512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:06.353440    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:06.710517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:06.710473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:06.937436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:06.937127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:07.171086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:07.171049    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:07.171649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:07.171115    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:08.021180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.020140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:08.152993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.152960    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:08.158925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.158894    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.170123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.170095    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 04:49:08.170286 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.170148    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 04:49:08.177140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.177114    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:08.186486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.185202    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:49:08.186486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.185378    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xtrvj\""
Oct 31 04:49:08.186486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.185485    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-446b6997e522\""
Oct 31 04:49:08.186486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.186129    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:49:08.187791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.187764    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:49:08.187791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.187786    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:49:08.239094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.239054    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.239094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.239101    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.239388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.239279    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf585\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.239388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.239381    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.354465    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.353619    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.354582    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.354665    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf585\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.354732    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.360843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.359411    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.366941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.366908    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.393385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.393304    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf585\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\") pod \"payload-processing-external-model-85f4bbf964-9kp5p\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") " pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.474597 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.474555    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:08.555393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.554738    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:08.876658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:08.876631    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:08.880750 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:49:08.880693    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae8b037_244e_4578_ab16_999e34804f82.slice/crio-8b43ced7a53eac7dc222506b5feaae59b04d0078ddb167dfb1521c57755853f4 WatchSource:0}: Error finding container 8b43ced7a53eac7dc222506b5feaae59b04d0078ddb167dfb1521c57755853f4: Status 404 returned error can't find the container with id 8b43ced7a53eac7dc222506b5feaae59b04d0078ddb167dfb1521c57755853f4
Oct 31 04:49:09.066992 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.066955    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" event={"ID":"0ae8b037-244e-4578-ab16-999e34804f82","Type":"ContainerStarted","Data":"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"}
Oct 31 04:49:09.067193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.067018    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" event={"ID":"0ae8b037-244e-4578-ab16-999e34804f82","Type":"ContainerStarted","Data":"8b43ced7a53eac7dc222506b5feaae59b04d0078ddb167dfb1521c57755853f4"}
Oct 31 04:49:09.067193 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.067107    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:09.110453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.110389    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" podStartSLOduration=1.110375336 podStartE2EDuration="1.110375336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:09.106709146 +0000 UTC m=+2645.139708841" watchObservedRunningTime="2026-10-31 04:49:09.110375336 +0000 UTC m=+2645.143375011"
Oct 31 04:49:09.665537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.665507    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:09.666114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.665992    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:09.666866 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:09.666846    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 04:49:10.414945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:10.414913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:10.759176 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:49:10.759071    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:49:12.185541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:12.185503    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:13.121687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:13.121655    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:13.124820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:13.124796    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:13.258962 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:13.258918    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:13.475268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:13.475173    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:14.297678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:14.297625    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:15.024125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:15.024099    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:49:16.124013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:16.123966    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:16.345461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:16.345424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:16.556225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:16.556188    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:16.778142 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:16.777684    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:17.499873 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:17.499823    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:17.740226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:17.736499    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:18.122682 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.122636    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" podUID="0ae8b037-244e-4578-ab16-999e34804f82" containerName="payload-processing" containerID="cri-o://11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189" gracePeriod=25
Oct 31 04:49:18.595071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.595040    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:18.678778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.678742    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\") pod \"0ae8b037-244e-4578-ab16-999e34804f82\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") "
Oct 31 04:49:18.678972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.678807    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\") pod \"0ae8b037-244e-4578-ab16-999e34804f82\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") "
Oct 31 04:49:18.678972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.678903    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\") pod \"0ae8b037-244e-4578-ab16-999e34804f82\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") "
Oct 31 04:49:18.679111 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.678986    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\") pod \"0ae8b037-244e-4578-ab16-999e34804f82\" (UID: \"0ae8b037-244e-4578-ab16-999e34804f82\") "
Oct 31 04:49:18.683358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.683292    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume" pod "0ae8b037-244e-4578-ab16-999e34804f82" (UID: "0ae8b037-244e-4578-ab16-999e34804f82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:18.701218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.686891    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume" pod "0ae8b037-244e-4578-ab16-999e34804f82" (UID: "0ae8b037-244e-4578-ab16-999e34804f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:18.722450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.710729    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585" pod "0ae8b037-244e-4578-ab16-999e34804f82" (UID: "0ae8b037-244e-4578-ab16-999e34804f82"). InnerVolumeSpecName "kube-api-access-pf585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:18.722450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.722338    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials" pod "0ae8b037-244e-4578-ab16-999e34804f82" (UID: "0ae8b037-244e-4578-ab16-999e34804f82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:18.797753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.797653    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:49:18.797753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.797692    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf585\" (UniqueName: \"kubernetes.io/projected/0ae8b037-244e-4578-ab16-999e34804f82-kube-api-access-pf585\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:49:18.797753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.797707    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:49:18.797753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:18.797720    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ae8b037-244e-4578-ab16-999e34804f82-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:49:19.120906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.120858    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:19.206848 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.206805    6931 generic.go:356] "Generic (PLEG): container finished" podID="0ae8b037-244e-4578-ab16-999e34804f82" containerID="11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189" exitCode=0
Oct 31 04:49:19.207063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.207004    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"
Oct 31 04:49:19.207063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.207010    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" event={"ID":"0ae8b037-244e-4578-ab16-999e34804f82","Type":"ContainerDied","Data":"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"}
Oct 31 04:49:19.207149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.207085    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p" event={"ID":"0ae8b037-244e-4578-ab16-999e34804f82","Type":"ContainerDied","Data":"8b43ced7a53eac7dc222506b5feaae59b04d0078ddb167dfb1521c57755853f4"}
Oct 31 04:49:19.207149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.207117    6931 scope.go:122] "RemoveContainer" containerID="11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"
Oct 31 04:49:19.220952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.220907    6931 scope.go:122] "RemoveContainer" containerID="11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"
Oct 31 04:49:19.221773 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:49:19.221743    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189\": container with ID starting with 11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189 not found: ID does not exist" containerID="11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"
Oct 31 04:49:19.221935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.221787    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189"} err="failed to get container status \"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189\": rpc error: code = NotFound desc = could not find container \"11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189\": container with ID starting with 11ea7d1fc1374f4496300562388029062038ef415d4be0a00c4d1122633d7189 not found: ID does not exist"
Oct 31 04:49:19.261585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.261550    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:19.273037 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.273006    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85f4bbf964-9kp5p"]
Oct 31 04:49:19.338549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.338513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:19.539225 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.539192    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:49:19.539834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.539820    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0ae8b037-244e-4578-ab16-999e34804f82" containerName="payload-processing"
Oct 31 04:49:19.539875 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.539835    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae8b037-244e-4578-ab16-999e34804f82" containerName="payload-processing"
Oct 31 04:49:19.539934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.539925    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae8b037-244e-4578-ab16-999e34804f82" containerName="payload-processing"
Oct 31 04:49:19.544699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.544683    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.562233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.562203    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"agc-cred-0d8ff4e65b60\""
Oct 31 04:49:19.562233 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.562203    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-e2e--201094b0-dockercfg-mbjzh\""
Oct 31 04:49:19.563332 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.563309    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:49:19.571367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.571309    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"payload-processing-external-model-plugins-e2e-extmod-a-401761\""
Oct 31 04:49:19.571367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.571316    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"openshift-service-ca.crt\""
Oct 31 04:49:19.571367 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.571319    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"routing-overlay\""
Oct 31 04:49:19.571575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.571388    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"kube-root-ca.crt\""
Oct 31 04:49:19.608212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.608174    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.608621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.608235    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvmq\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.608621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.608277    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.608621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.608424    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.736045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.736003    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.736256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.736068    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvmq\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.736256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.736116    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.736342 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.736307    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.737830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.737799    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.740340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.738978    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.741061 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.741015    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.764222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.764051    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvmq\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:19.859630 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:19.859534    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:20.123260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:20.123189    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae8b037-244e-4578-ab16-999e34804f82" path="/var/lib/kubelet/pods/0ae8b037-244e-4578-ab16-999e34804f82/volumes"
Oct 31 04:49:20.165388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:20.165345    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:49:20.217239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:20.217193    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" event={"ID":"04ed23c3-badb-455a-b982-2309ca2adbfb","Type":"ContainerStarted","Data":"a8a7647e0b6b1e16e8a4dbd2e68793dde3138ec03084a6bd7d765e554d4154c9"}
Oct 31 04:49:20.745176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:20.743208    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:21.206241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:21.206208    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:21.229168 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:21.229124    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" event={"ID":"04ed23c3-badb-455a-b982-2309ca2adbfb","Type":"ContainerStarted","Data":"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"}
Oct 31 04:49:21.229379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:21.229317    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:21.271748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:21.271691    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" podStartSLOduration=2.271677549 podStartE2EDuration="2.271677549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:49:21.269300611 +0000 UTC m=+2657.302300290" watchObservedRunningTime="2026-10-31 04:49:21.271677549 +0000 UTC m=+2657.304677265"
Oct 31 04:49:22.922935 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:22.922892    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:23.132145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:23.132108    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:25.017327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:25.017287    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:26.828985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:26.828932    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:28.724126 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:28.724091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:30.515012 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:30.514976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:32.252923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:32.252509    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:49:32.285258 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:32.285225    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:34.036708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:34.036666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:35.886924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:35.886850    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:37.710574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:37.710539    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:39.510683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:39.510599    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:40.759843 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:49:40.759803    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:49:41.265834 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:41.265796    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:42.965550 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:42.965459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:44.716780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:44.716743    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:46.581770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:46.581733    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:48.533460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:48.533416    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:50.407554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:50.407505    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:52.361463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:52.361420    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:54.201963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:54.201922    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:56.052386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:56.052344    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:57.866331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:57.866290    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:59.683606 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:49:59.683562    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:01.031432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:01.031394    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:01.712686 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:01.712647    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:03.418841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:03.418652    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:05.205289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:05.205257    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:07.094433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:07.094394    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:08.905384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:08.905243    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:10.702816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:10.702778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:10.760222 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:50:10.760186    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:50:11.883819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:11.883777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:12.098864 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:12.098826    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:12.498916 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:12.498876    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:12.862560 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:12.862520    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:14.671804 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:14.671714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:16.491615 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:16.491575    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:18.385923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:18.385819    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:18.972434 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:18.972390    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:19.193835 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:19.193796    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:20.223257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:20.223224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:22.106255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:22.106223    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:23.826284 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:23.826234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:25.409839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:25.409804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:25.636180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:25.633033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:25.847618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:25.847581    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:26.064666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:26.064626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:26.279213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:26.279124    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:26.835092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:26.835051    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:27.110236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:27.109688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:28.016972 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:28.016936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:28.525724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:28.525688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:29.626823 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:29.626071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:29.961923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:29.961775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:31.524180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:31.517983    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:31.800537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:31.800486    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:32.011253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:32.011214    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:32.219223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:32.219125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:32.995865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:32.995819    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:33.207686 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:33.207646    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:34.056731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.056694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:34.272867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.272831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:34.423092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.423015    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:50:34.427298 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.427278    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.446745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.446720    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:50:34.454914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.454888    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-654a74d9dcec\""
Oct 31 04:50:34.455088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.454906    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:50:34.455088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.454916    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:50:34.455088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.454970    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:50:34.455294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.454980    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-g8d4f\""
Oct 31 04:50:34.455294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.455036    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:50:34.533197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.533126    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5jl\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.533407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.533220    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.533407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.533335    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.533407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.533395    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.637178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.637082    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.637378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.637184    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.637378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.637229    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.637500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.637418    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5jl\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.640207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.639270    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.640207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.639906    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.665824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.665792    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.674417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.674333    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5jl\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\") pod \"payload-processing-external-model-57d6774b9c-pzf46\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") " pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:34.742398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:34.742326    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:35.072585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.072556    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:50:35.075747 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:50:35.075721    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933169fb_4692_4af0_b290_a9f1f1fff087.slice/crio-617ce0409a5352a7c38680d0b2347ac40c4291b55c8719c8799e7838b5742bcf WatchSource:0}: Error finding container 617ce0409a5352a7c38680d0b2347ac40c4291b55c8719c8799e7838b5742bcf: Status 404 returned error can't find the container with id 617ce0409a5352a7c38680d0b2347ac40c4291b55c8719c8799e7838b5742bcf
Oct 31 04:50:35.774321 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.774282    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" event={"ID":"933169fb-4692-4af0-b290-a9f1f1fff087","Type":"ContainerStarted","Data":"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"}
Oct 31 04:50:35.774505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.774359    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" event={"ID":"933169fb-4692-4af0-b290-a9f1f1fff087","Type":"ContainerStarted","Data":"617ce0409a5352a7c38680d0b2347ac40c4291b55c8719c8799e7838b5742bcf"}
Oct 31 04:50:35.774505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.774448    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:35.817568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.817500    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" podStartSLOduration=1.817477762 podStartE2EDuration="1.817477762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:50:35.816031058 +0000 UTC m=+2731.849030738" watchObservedRunningTime="2026-10-31 04:50:35.817477762 +0000 UTC m=+2731.850477443"
Oct 31 04:50:35.946022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:35.943056    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:37.823641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:37.823600    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:39.595467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:39.595429    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:40.758920 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:50:40.758878    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:50:41.441179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:41.434444    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:43.229732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:43.229696    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:45.005558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:45.005512    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:46.737821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:46.737786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:46.783514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:46.783488    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:50:48.545049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:48.545012    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:50.368101 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:50.368032    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:52.116825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:52.116777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:54.662136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:54.662101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:55.722325 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:55.722284    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:50:55.722757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:55.722353    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:50:56.919629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:56.919598    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:58.909376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:50:58.909334    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:00.913264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:00.913171    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:01.190596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:01.190509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:02.349504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:02.349471    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:02.566907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:02.566873    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:03.040137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.040101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:03.757957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.757870    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:03.897253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.897217    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:51:03.901371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.901345    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930244    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-plugins-e2e-extmod-b-227c93\""
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930324    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"agc-cred-759e21c40052\""
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930350    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-e2e--299d87ac-dockercfg-zdmh8\""
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930411    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"kube-root-ca.crt\""
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930459    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"openshift-service-ca.crt\""
Oct 31 04:51:03.930784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.930485    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"routing-overlay\""
Oct 31 04:51:03.935095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:03.935059    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:51:04.117472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.117305    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbn9\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.117907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.117872    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.119176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.118594    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.119472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.119452    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.220569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.220530    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbn9\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.220769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.220600    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.220769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.220668    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.220881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.220774    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.234261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.234232    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"agc-cred-759e21c40052\""
Oct 31 04:51:04.234261 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.234250    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"routing-overlay\""
Oct 31 04:51:04.234458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.234284    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-plugins-e2e-extmod-b-227c93\""
Oct 31 04:51:04.241527 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.241504    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.241654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.241628    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.243118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.243101    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.254147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.254126    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"kube-root-ca.crt\""
Oct 31 04:51:04.274436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.274402    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"openshift-service-ca.crt\""
Oct 31 04:51:04.284183 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.284133    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbn9\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.530706 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.530675    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-e2e--299d87ac-dockercfg-zdmh8\""
Oct 31 04:51:04.537696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.537672    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:04.811294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.811202    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:04.846465 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:04.846441    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:51:04.849519 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:51:04.849491    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1734993c_a11c_407e_8e4b_4fc5fba766f0.slice/crio-4f7349f6e6dd8c11c4746359cfcbfc91422108545bf2ac6da9ef9b8e30dce9d9 WatchSource:0}: Error finding container 4f7349f6e6dd8c11c4746359cfcbfc91422108545bf2ac6da9ef9b8e30dce9d9: Status 404 returned error can't find the container with id 4f7349f6e6dd8c11c4746359cfcbfc91422108545bf2ac6da9ef9b8e30dce9d9
Oct 31 04:51:05.118922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:05.118829    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" event={"ID":"1734993c-a11c-407e-8e4b-4fc5fba766f0","Type":"ContainerStarted","Data":"ff68efa55599d19a7546128fd903158ddb6ca35c6602ec6972a645d03d912723"}
Oct 31 04:51:05.118922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:05.118885    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" event={"ID":"1734993c-a11c-407e-8e4b-4fc5fba766f0","Type":"ContainerStarted","Data":"4f7349f6e6dd8c11c4746359cfcbfc91422108545bf2ac6da9ef9b8e30dce9d9"}
Oct 31 04:51:05.119128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:05.118925    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:05.160070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:05.160018    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" podStartSLOduration=2.160003775 podStartE2EDuration="2.160003775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:05.159597489 +0000 UTC m=+2761.192597160" watchObservedRunningTime="2026-10-31 04:51:05.160003775 +0000 UTC m=+2761.193003454"
Oct 31 04:51:06.603242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:06.603201    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:08.412283 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:08.412241    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:10.224000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:10.223945    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:10.759822 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:51:10.759787    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:51:12.191026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:12.190990    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:12.684667 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:12.684616    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:51:12.832176 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:12.832126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:13.072198 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:13.066081    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:14.312958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:14.312917    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:14.527368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:14.527332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:14.762507 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:14.762403    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:15.040309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:15.040277    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:15.623620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:15.623569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:15.830577 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:15.830537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:16.127770 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:16.127740    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:51:16.453247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:16.441131    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:17.220222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:17.220111    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:17.685582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:17.685534    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" podUID="933169fb-4692-4af0-b290-a9f1f1fff087" containerName="payload-processing" containerID="cri-o://e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c" gracePeriod=25
Oct 31 04:51:17.959345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:17.959322    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:51:18.031716 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.031678    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\") pod \"933169fb-4692-4af0-b290-a9f1f1fff087\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") "
Oct 31 04:51:18.031912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.031742    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\") pod \"933169fb-4692-4af0-b290-a9f1f1fff087\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") "
Oct 31 04:51:18.031912 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.031845    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\") pod \"933169fb-4692-4af0-b290-a9f1f1fff087\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") "
Oct 31 04:51:18.032041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.031926    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\") pod \"933169fb-4692-4af0-b290-a9f1f1fff087\" (UID: \"933169fb-4692-4af0-b290-a9f1f1fff087\") "
Oct 31 04:51:18.032322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.032290    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume" pod "933169fb-4692-4af0-b290-a9f1f1fff087" (UID: "933169fb-4692-4af0-b290-a9f1f1fff087"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:18.032657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.032632    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:18.032737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.032669    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume" pod "933169fb-4692-4af0-b290-a9f1f1fff087" (UID: "933169fb-4692-4af0-b290-a9f1f1fff087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:18.034373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.034343    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials" pod "933169fb-4692-4af0-b290-a9f1f1fff087" (UID: "933169fb-4692-4af0-b290-a9f1f1fff087"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:18.034522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.034500    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl" pod "933169fb-4692-4af0-b290-a9f1f1fff087" (UID: "933169fb-4692-4af0-b290-a9f1f1fff087"). InnerVolumeSpecName "kube-api-access-5w5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:18.133510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.133481    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:18.133510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.133506    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933169fb-4692-4af0-b290-a9f1f1fff087-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:18.133510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.133515    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5jl\" (UniqueName: \"kubernetes.io/projected/933169fb-4692-4af0-b290-a9f1f1fff087-kube-api-access-5w5jl\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:18.268528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.268434    6931 generic.go:356] "Generic (PLEG): container finished" podID="933169fb-4692-4af0-b290-a9f1f1fff087" containerID="e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c" exitCode=0
Oct 31 04:51:18.268528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.268491    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" event={"ID":"933169fb-4692-4af0-b290-a9f1f1fff087","Type":"ContainerDied","Data":"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"}
Oct 31 04:51:18.268528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.268496    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"
Oct 31 04:51:18.269235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.268533    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46" event={"ID":"933169fb-4692-4af0-b290-a9f1f1fff087","Type":"ContainerDied","Data":"617ce0409a5352a7c38680d0b2347ac40c4291b55c8719c8799e7838b5742bcf"}
Oct 31 04:51:18.269235 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.268554    6931 scope.go:122] "RemoveContainer" containerID="e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"
Oct 31 04:51:18.279097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.279068    6931 scope.go:122] "RemoveContainer" containerID="e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"
Oct 31 04:51:18.279413 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:51:18.279387    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c\": container with ID starting with e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c not found: ID does not exist" containerID="e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"
Oct 31 04:51:18.279474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.279422    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c"} err="failed to get container status \"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c\": rpc error: code = NotFound desc = could not find container \"e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c\": container with ID starting with e83105be679a03c6737b3340df3ab8efc868a34e328f1062a90e178f5a1ae93c not found: ID does not exist"
Oct 31 04:51:18.332981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.319112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:18.340218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.333919    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:51:18.340495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.340435    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d6774b9c-pzf46"]
Oct 31 04:51:18.466292 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.466258    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:18.466950 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.466934    6931 cpu_manager.go:414] "containerMap: removing container" podUID="933169fb-4692-4af0-b290-a9f1f1fff087" containerName="payload-processing"
Oct 31 04:51:18.467000 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.466954    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933169fb-4692-4af0-b290-a9f1f1fff087" containerName="payload-processing"
Oct 31 04:51:18.467094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.467084    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="933169fb-4692-4af0-b290-a9f1f1fff087" containerName="payload-processing"
Oct 31 04:51:18.472118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.472093    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.489494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.489465    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"agc-cred-d2374fac1bc0\""
Oct 31 04:51:18.490114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.490093    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:18.492350 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.492331    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-e2e--6f807998-dockercfg-zgf86\""
Oct 31 04:51:18.492744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.492721    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"routing-overlay\""
Oct 31 04:51:18.492861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.492745    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"kube-root-ca.crt\""
Oct 31 04:51:18.492861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.492842    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"payload-processing-external-model-plugins-e2e-extmod-a-47c409\""
Oct 31 04:51:18.492861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.492842    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"openshift-service-ca.crt\""
Oct 31 04:51:18.550906 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.550865    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.551094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.550982    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.551144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.551111    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.551254 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.551234    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7d9\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.659082    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.659221    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.659358    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.659472    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7d9\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.660649    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.661639    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.687179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.686566    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:18.696201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.691548    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7d9\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.703523 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.703458    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:18.786132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:18.786092    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:19.112242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:19.112141    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:19.116378 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:51:19.116352    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda427413d_9513_4374_ad43_87a21e88aa69.slice/crio-4abbe3b3768a41e78f1c5607dc357681f0d97cef664847f4eecefe8055ae03c2 WatchSource:0}: Error finding container 4abbe3b3768a41e78f1c5607dc357681f0d97cef664847f4eecefe8055ae03c2: Status 404 returned error can't find the container with id 4abbe3b3768a41e78f1c5607dc357681f0d97cef664847f4eecefe8055ae03c2
Oct 31 04:51:19.278021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:19.277980    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" event={"ID":"a427413d-9513-4374-ad43-87a21e88aa69","Type":"ContainerStarted","Data":"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"}
Oct 31 04:51:19.278536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:19.278043    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" event={"ID":"a427413d-9513-4374-ad43-87a21e88aa69","Type":"ContainerStarted","Data":"4abbe3b3768a41e78f1c5607dc357681f0d97cef664847f4eecefe8055ae03c2"}
Oct 31 04:51:19.278536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:19.278193    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:19.332884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:19.332820    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" podStartSLOduration=1.33280194 podStartE2EDuration="1.33280194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:19.330009053 +0000 UTC m=+2775.363008733" watchObservedRunningTime="2026-10-31 04:51:19.33280194 +0000 UTC m=+2775.365801623"
Oct 31 04:51:20.122739 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:20.122704    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933169fb-4692-4af0-b290-a9f1f1fff087" path="/var/lib/kubelet/pods/933169fb-4692-4af0-b290-a9f1f1fff087/volumes"
Oct 31 04:51:20.436847 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:20.436748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:22.218037 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:22.217996    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:22.435093 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:22.435050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:22.729841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:22.729806    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:23.677824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:23.677767    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:23.883051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:23.883012    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:24.274928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:24.274890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:24.684655 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:24.684606    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:51:24.801570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:24.801530    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:25.020112 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:25.020022    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:25.400439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:25.400395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:26.297069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:26.297033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:27.678215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.678185    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:51:27.682921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.682902    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.700942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.700913    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:51:27.708732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.708705    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-f5a14c\"/\"openshift-service-ca.crt\""
Oct 31 04:51:27.709247 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.709219    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-f5a14c\"/\"e2e-usage-sim-04e0c1-cert\""
Oct 31 04:51:27.709361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.709269    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-f5a14c\"/\"default-dockercfg-nd4pj\""
Oct 31 04:51:27.709361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.709314    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-f5a14c\"/\"kube-root-ca.crt\""
Oct 31 04:51:27.802977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.802938    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.802977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.802981    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh65\" (UniqueName: \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.905025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.904994    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.905187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.905033    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh65\" (UniqueName: \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.907479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.907454    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.928102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.928069    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh65\" (UniqueName: \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\") pod \"e2e-usage-sim-04e0c1-6955658f4c-5ws64\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") " pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:27.998232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:27.996432    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:28.041776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:28.041706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:28.325091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:28.325051    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:51:28.329735 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:51:28.329702    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee0da6dd_8056_4b11_9236_89736c2e809f.slice/crio-90198e782f83b54af73f3cd2b431e521e11ae95d49afbc8161a262e742e7bd67 WatchSource:0}: Error finding container 90198e782f83b54af73f3cd2b431e521e11ae95d49afbc8161a262e742e7bd67: Status 404 returned error can't find the container with id 90198e782f83b54af73f3cd2b431e521e11ae95d49afbc8161a262e742e7bd67
Oct 31 04:51:28.357416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:28.357386    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" event={"ID":"ee0da6dd-8056-4b11-9236-89736c2e809f","Type":"ContainerStarted","Data":"90198e782f83b54af73f3cd2b431e521e11ae95d49afbc8161a262e742e7bd67"}
Oct 31 04:51:29.364888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.364841    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" event={"ID":"ee0da6dd-8056-4b11-9236-89736c2e809f","Type":"ContainerStarted","Data":"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"}
Oct 31 04:51:29.364888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.364894    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:29.369975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.369949    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:51:29.405255 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.405195    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" podStartSLOduration=2.405151887 podStartE2EDuration="2.405151887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:29.404957373 +0000 UTC m=+2785.437957065" watchObservedRunningTime="2026-10-31 04:51:29.405151887 +0000 UTC m=+2785.438151567"
Oct 31 04:51:29.685066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.684985    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" podUID="46225329-5550-46d2-9bfb-7b9218f34640" containerName="payload-processing" containerID="cri-o://1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66" gracePeriod=25
Oct 31 04:51:29.836587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.836506    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:29.975229 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:29.975204    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:51:30.042360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.042321    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\") pod \"46225329-5550-46d2-9bfb-7b9218f34640\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") "
Oct 31 04:51:30.042559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.042401    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\") pod \"46225329-5550-46d2-9bfb-7b9218f34640\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") "
Oct 31 04:51:30.042636 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.042594    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\") pod \"46225329-5550-46d2-9bfb-7b9218f34640\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") "
Oct 31 04:51:30.042776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.042755    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\") pod \"46225329-5550-46d2-9bfb-7b9218f34640\" (UID: \"46225329-5550-46d2-9bfb-7b9218f34640\") "
Oct 31 04:51:30.042857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.042797    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume" pod "46225329-5550-46d2-9bfb-7b9218f34640" (UID: "46225329-5550-46d2-9bfb-7b9218f34640"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:30.043271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.043233    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume" pod "46225329-5550-46d2-9bfb-7b9218f34640" (UID: "46225329-5550-46d2-9bfb-7b9218f34640"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:30.043365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.043340    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:30.043413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.043363    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46225329-5550-46d2-9bfb-7b9218f34640-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:30.045199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.045140    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials" pod "46225329-5550-46d2-9bfb-7b9218f34640" (UID: "46225329-5550-46d2-9bfb-7b9218f34640"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:30.045199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.045185    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2" pod "46225329-5550-46d2-9bfb-7b9218f34640" (UID: "46225329-5550-46d2-9bfb-7b9218f34640"). InnerVolumeSpecName "kube-api-access-pkxk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:30.144446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.144406    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:30.144446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.144441    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxk2\" (UniqueName: \"kubernetes.io/projected/46225329-5550-46d2-9bfb-7b9218f34640-kube-api-access-pkxk2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:30.288631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.288599    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:30.372062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.372029    6931 generic.go:356] "Generic (PLEG): container finished" podID="46225329-5550-46d2-9bfb-7b9218f34640" containerID="1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66" exitCode=0
Oct 31 04:51:30.372661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.372077    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"
Oct 31 04:51:30.372661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.372119    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" event={"ID":"46225329-5550-46d2-9bfb-7b9218f34640","Type":"ContainerDied","Data":"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"}
Oct 31 04:51:30.372661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.372199    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8" event={"ID":"46225329-5550-46d2-9bfb-7b9218f34640","Type":"ContainerDied","Data":"08d86f14441eb4370e9fdab762e8ba608df3e3445d04ffd9a72ac2735c504118"}
Oct 31 04:51:30.372661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.372233    6931 scope.go:122] "RemoveContainer" containerID="1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"
Oct 31 04:51:30.382680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.382658    6931 scope.go:122] "RemoveContainer" containerID="1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"
Oct 31 04:51:30.382989 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:51:30.382963    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66\": container with ID starting with 1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66 not found: ID does not exist" containerID="1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"
Oct 31 04:51:30.383064 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.382988    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66"} err="failed to get container status \"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66\": rpc error: code = NotFound desc = could not find container \"1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66\": container with ID starting with 1c04d11b5d3f801dd0398b11c05cab276d6ca82fa9a840a3c275fcc074926e66 not found: ID does not exist"
Oct 31 04:51:30.424079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.424042    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:51:30.428214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:30.428186    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-afc519/payload-processing-external-model-e2e-extmod-a-afc519-8496ckfj8"]
Oct 31 04:51:31.429726 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:31.429688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:31.637620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:31.637586    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:31.845787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:31.845754    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:32.123713 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:32.123638    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46225329-5550-46d2-9bfb-7b9218f34640" path="/var/lib/kubelet/pods/46225329-5550-46d2-9bfb-7b9218f34640/volumes"
Oct 31 04:51:33.532920 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:33.532887    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:33.775849 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:33.775796    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:34.006230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:34.003935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:35.814536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:35.814472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:36.340996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.340944    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:51:36.341728 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.341708    6931 cpu_manager.go:414] "containerMap: removing container" podUID="46225329-5550-46d2-9bfb-7b9218f34640" containerName="payload-processing"
Oct 31 04:51:36.341841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.341729    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46225329-5550-46d2-9bfb-7b9218f34640" containerName="payload-processing"
Oct 31 04:51:36.341914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.341899    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="46225329-5550-46d2-9bfb-7b9218f34640" containerName="payload-processing"
Oct 31 04:51:36.345640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.345620    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.361018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.360979    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-47c409\"/\"agc-cred-ab3a1b566ecf\""
Oct 31 04:51:36.365041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.365018    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:51:36.448035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.447998    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.448248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.448134    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pvv\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.448248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.448227    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.448375 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.448344    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.549968    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pvv\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.550012    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.550064    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550346 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.550292    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550829 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.550804    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.550942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.550924    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.552585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.552568    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.572563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.572538    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pvv\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-5f467949k\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.659512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.659424    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:36.968802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:36.968754    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:51:36.972084 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:51:36.972059    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8391ba07_9a69_4e6f_9b8d_44a63069334f.slice/crio-18dff8e69c20b930638a078b2a653f2c9ec82e363292047c9397e8f76e5c7d61 WatchSource:0}: Error finding container 18dff8e69c20b930638a078b2a653f2c9ec82e363292047c9397e8f76e5c7d61: Status 404 returned error can't find the container with id 18dff8e69c20b930638a078b2a653f2c9ec82e363292047c9397e8f76e5c7d61
Oct 31 04:51:37.421462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:37.421424    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" event={"ID":"8391ba07-9a69-4e6f-9b8d-44a63069334f","Type":"ContainerStarted","Data":"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"}
Oct 31 04:51:37.421750 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:37.421489    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" event={"ID":"8391ba07-9a69-4e6f-9b8d-44a63069334f","Type":"ContainerStarted","Data":"18dff8e69c20b930638a078b2a653f2c9ec82e363292047c9397e8f76e5c7d61"}
Oct 31 04:51:37.421838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:37.421779    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:37.463459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:37.463405    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" podStartSLOduration=1.463390121 podStartE2EDuration="1.463390121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:37.461379124 +0000 UTC m=+2793.494378803" watchObservedRunningTime="2026-10-31 04:51:37.463390121 +0000 UTC m=+2793.496389799"
Oct 31 04:51:37.717578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:37.717487    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:39.183079 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:39.183033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:39.403210 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:39.399606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:39.828632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:39.828590    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:40.169230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.168925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:40.329417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.329384    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:51:40.333546 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.333525    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.361670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.361639    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:51:40.362429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.361639    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:51:40.362429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.361659    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-494v4\""
Oct 31 04:51:40.362429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.361675    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:51:40.362429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.361805    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ca6201104fc9\""
Oct 31 04:51:40.362429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.362031    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:51:40.365419 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.365390    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:51:40.420029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.419943    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.420029 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.420018    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.420294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.420114    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdbw\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.420360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.420309    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.523478    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.523664    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.523726    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.523753    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdbw\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.524987    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.527206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.525935    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.530014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.529963    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.556248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.556211    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdbw\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\") pod \"payload-processing-external-model-7cf99f468c-xjg96\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") " pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.649495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.649457    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:40.770201 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:51:40.768343    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:51:40.964793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:40.964757    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:51:40.967991 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:51:40.967963    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6201d40_fdfe_45be_9075_65eb88e05139.slice/crio-06975d1fa96c0907172589c0a72b89c37333ef6685956625a5c1129ec376f4a5 WatchSource:0}: Error finding container 06975d1fa96c0907172589c0a72b89c37333ef6685956625a5c1129ec376f4a5: Status 404 returned error can't find the container with id 06975d1fa96c0907172589c0a72b89c37333ef6685956625a5c1129ec376f4a5
Oct 31 04:51:41.458723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:41.458685    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" event={"ID":"a6201d40-fdfe-45be-9075-65eb88e05139","Type":"ContainerStarted","Data":"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"}
Oct 31 04:51:41.459187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:41.458741    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" event={"ID":"a6201d40-fdfe-45be-9075-65eb88e05139","Type":"ContainerStarted","Data":"06975d1fa96c0907172589c0a72b89c37333ef6685956625a5c1129ec376f4a5"}
Oct 31 04:51:41.459187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:41.458768    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:41.501956 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:41.501697    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" podStartSLOduration=1.5016346889999999 podStartE2EDuration="1.501634689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:41.500589419 +0000 UTC m=+2797.533589111" watchObservedRunningTime="2026-10-31 04:51:41.501634689 +0000 UTC m=+2797.534634367"
Oct 31 04:51:41.839937 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:41.839894    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:43.806814 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:43.806776    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:45.605413 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:45.605337    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:47.408272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:47.408232    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:48.433036 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:48.433006    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:51:48.506725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:48.506692    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:49.171928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:49.171894    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:50.922031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:50.921994    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:52.468616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:52.468588    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:51:52.730510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:52.730428    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:53.507276 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.507230    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" podUID="a427413d-9513-4374-ad43-87a21e88aa69" containerName="payload-processing" containerID="cri-o://1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2" gracePeriod=25
Oct 31 04:51:53.776601 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.776533    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:53.808681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.808648    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\") pod \"a427413d-9513-4374-ad43-87a21e88aa69\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") "
Oct 31 04:51:53.808885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.808867    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\") pod \"a427413d-9513-4374-ad43-87a21e88aa69\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") "
Oct 31 04:51:53.809053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.809031    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\") pod \"a427413d-9513-4374-ad43-87a21e88aa69\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") "
Oct 31 04:51:53.809138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.809075    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\") pod \"a427413d-9513-4374-ad43-87a21e88aa69\" (UID: \"a427413d-9513-4374-ad43-87a21e88aa69\") "
Oct 31 04:51:53.809239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.809129    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume" pod "a427413d-9513-4374-ad43-87a21e88aa69" (UID: "a427413d-9513-4374-ad43-87a21e88aa69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:53.809509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.809483    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume" pod "a427413d-9513-4374-ad43-87a21e88aa69" (UID: "a427413d-9513-4374-ad43-87a21e88aa69"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:53.809589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.809534    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:53.811287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.811258    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9" pod "a427413d-9513-4374-ad43-87a21e88aa69" (UID: "a427413d-9513-4374-ad43-87a21e88aa69"). InnerVolumeSpecName "kube-api-access-dx7d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:53.811390 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.811308    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials" pod "a427413d-9513-4374-ad43-87a21e88aa69" (UID: "a427413d-9513-4374-ad43-87a21e88aa69"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:53.910886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.910840    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:53.910886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.910877    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a427413d-9513-4374-ad43-87a21e88aa69-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:53.910886 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:53.910887    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7d9\" (UniqueName: \"kubernetes.io/projected/a427413d-9513-4374-ad43-87a21e88aa69-kube-api-access-dx7d9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:51:54.513379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.513283    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:54.555684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.555648    6931 generic.go:356] "Generic (PLEG): container finished" podID="a427413d-9513-4374-ad43-87a21e88aa69" containerID="1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2" exitCode=0
Oct 31 04:51:54.555854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.555699    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"
Oct 31 04:51:54.555854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.555734    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" event={"ID":"a427413d-9513-4374-ad43-87a21e88aa69","Type":"ContainerDied","Data":"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"}
Oct 31 04:51:54.555854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.555787    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw" event={"ID":"a427413d-9513-4374-ad43-87a21e88aa69","Type":"ContainerDied","Data":"4abbe3b3768a41e78f1c5607dc357681f0d97cef664847f4eecefe8055ae03c2"}
Oct 31 04:51:54.555854 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.555810    6931 scope.go:122] "RemoveContainer" containerID="1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"
Oct 31 04:51:54.565306 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.565285    6931 scope.go:122] "RemoveContainer" containerID="1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"
Oct 31 04:51:54.565583 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:51:54.565561    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2\": container with ID starting with 1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2 not found: ID does not exist" containerID="1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"
Oct 31 04:51:54.565648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.565592    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2"} err="failed to get container status \"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2\": rpc error: code = NotFound desc = could not find container \"1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2\": container with ID starting with 1acfc7fdb6e156658379808286fc7f0be77f5f1bd0c95d6120a63a742ebbc6d2 not found: ID does not exist"
Oct 31 04:51:54.606821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.606786    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:54.618407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:54.618371    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c8nlnw"]
Oct 31 04:51:56.125354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:56.125319    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a427413d-9513-4374-ad43-87a21e88aa69" path="/var/lib/kubelet/pods/a427413d-9513-4374-ad43-87a21e88aa69/volumes"
Oct 31 04:51:56.410752 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:56.410664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:58.143137 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:58.143102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:59.934748 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:51:59.934715    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:01.317668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:01.317629    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:01.815329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:01.815285    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:03.606691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:03.606651    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:05.427053 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:05.426958    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:07.175532 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:07.175493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:09.011032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:09.010997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:10.790900 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:52:10.790854    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:52:10.822266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:10.822229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:12.621690 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:12.621655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:14.355819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:14.355779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:16.221148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:16.221114    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:18.318015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:18.317963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:20.688239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:20.688196    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:22.439559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:22.439511    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:24.259685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:24.259648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:25.996034 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:25.995997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:27.813563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:27.813520    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:29.713675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:29.713635    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:31.577874 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:31.577831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:33.322821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:33.322786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:35.117850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:35.117808    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:36.907575 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:36.907537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:38.742011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:38.741962    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:40.576219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:40.576181    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:40.758848 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:52:40.758808    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:52:41.134710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:41.134673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:41.349007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:41.348968    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:42.907283 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:42.907181    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:43.583919 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.583879    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:52:43.584571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.584556    6931 cpu_manager.go:414] "containerMap: removing container" podUID="a427413d-9513-4374-ad43-87a21e88aa69" containerName="payload-processing"
Oct 31 04:52:43.584625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.584572    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a427413d-9513-4374-ad43-87a21e88aa69" containerName="payload-processing"
Oct 31 04:52:43.584679 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.584670    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a427413d-9513-4374-ad43-87a21e88aa69" containerName="payload-processing"
Oct 31 04:52:43.588469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.588447    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.616024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.615978    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-401761\"/\"agc-cred-4e343ec49203\""
Oct 31 04:52:43.618958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.618930    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:52:43.726497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.726464    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.726674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.726506    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.726674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.726572    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.726674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.726625    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9qk\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.828339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.828302    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.828515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.828367    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.828515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.828423    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.828515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.828487    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9qk\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.829231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.829204    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.829315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.829275    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.830918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.830900    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.851984 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.851923    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9qk\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\") pod \"payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:43.902674 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:43.902643    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:44.209988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.209916    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:52:44.213797 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:52:44.213772    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de01a64_bc2a_4272_b68c_b2191c0ec8c2.slice/crio-15eeef149dbcfeb78e3cf365aca7b17911d4d585dfbfd6931b24984edfe2e459 WatchSource:0}: Error finding container 15eeef149dbcfeb78e3cf365aca7b17911d4d585dfbfd6931b24984edfe2e459: Status 404 returned error can't find the container with id 15eeef149dbcfeb78e3cf365aca7b17911d4d585dfbfd6931b24984edfe2e459
Oct 31 04:52:44.699999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.699954    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:44.920684 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.920647    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" event={"ID":"1de01a64-bc2a-4272-b68c-b2191c0ec8c2","Type":"ContainerStarted","Data":"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"}
Oct 31 04:52:44.920860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.920703    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" event={"ID":"1de01a64-bc2a-4272-b68c-b2191c0ec8c2","Type":"ContainerStarted","Data":"15eeef149dbcfeb78e3cf365aca7b17911d4d585dfbfd6931b24984edfe2e459"}
Oct 31 04:52:44.920860 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.920741    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:44.963767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:44.963667    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" podStartSLOduration=1.963649152 podStartE2EDuration="1.963649152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:52:44.961416296 +0000 UTC m=+2860.994415976" watchObservedRunningTime="2026-10-31 04:52:44.963649152 +0000 UTC m=+2860.996648830"
Oct 31 04:52:46.513537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:46.513495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:48.253631 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:48.253593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:50.012677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:50.012633    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:51.798528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:51.798473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:53.679169 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:53.679123    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:55.434266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:55.434222    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:55.930280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:55.930244    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:52:56.001105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:56.001069    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:52:57.167762 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:57.167724    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:58.996149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:52:58.996110    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:00.805998 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:00.805958    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:01.002446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.002402    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" podUID="04ed23c3-badb-455a-b982-2309ca2adbfb" containerName="payload-processing" containerID="cri-o://6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2" gracePeriod=25
Oct 31 04:53:01.269368 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.269344    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:53:01.392374 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392291    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\") pod \"04ed23c3-badb-455a-b982-2309ca2adbfb\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") "
Oct 31 04:53:01.392563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392423    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\") pod \"04ed23c3-badb-455a-b982-2309ca2adbfb\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") "
Oct 31 04:53:01.392563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392490    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\") pod \"04ed23c3-badb-455a-b982-2309ca2adbfb\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") "
Oct 31 04:53:01.392694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392564    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\") pod \"04ed23c3-badb-455a-b982-2309ca2adbfb\" (UID: \"04ed23c3-badb-455a-b982-2309ca2adbfb\") "
Oct 31 04:53:01.392898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392760    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume" pod "04ed23c3-badb-455a-b982-2309ca2adbfb" (UID: "04ed23c3-badb-455a-b982-2309ca2adbfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:01.393032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.392911    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume" pod "04ed23c3-badb-455a-b982-2309ca2adbfb" (UID: "04ed23c3-badb-455a-b982-2309ca2adbfb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:01.393132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.393116    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:01.393204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.393134    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ed23c3-badb-455a-b982-2309ca2adbfb-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:01.394639 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.394617    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials" pod "04ed23c3-badb-455a-b982-2309ca2adbfb" (UID: "04ed23c3-badb-455a-b982-2309ca2adbfb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:01.394722 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.394637    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq" pod "04ed23c3-badb-455a-b982-2309ca2adbfb" (UID: "04ed23c3-badb-455a-b982-2309ca2adbfb"). InnerVolumeSpecName "kube-api-access-jpvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:01.438685 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.438626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:01.494231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.494193    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvmq\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-kube-api-access-jpvmq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:01.494231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:01.494226    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04ed23c3-badb-455a-b982-2309ca2adbfb-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:02.033621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.033579    6931 generic.go:356] "Generic (PLEG): container finished" podID="04ed23c3-badb-455a-b982-2309ca2adbfb" containerID="6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2" exitCode=0
Oct 31 04:53:02.033621 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.033614    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" event={"ID":"04ed23c3-badb-455a-b982-2309ca2adbfb","Type":"ContainerDied","Data":"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"}
Oct 31 04:53:02.034123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.033679    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6" event={"ID":"04ed23c3-badb-455a-b982-2309ca2adbfb","Type":"ContainerDied","Data":"a8a7647e0b6b1e16e8a4dbd2e68793dde3138ec03084a6bd7d765e554d4154c9"}
Oct 31 04:53:02.034123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.033626    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"
Oct 31 04:53:02.034123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.033710    6931 scope.go:122] "RemoveContainer" containerID="6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"
Oct 31 04:53:02.048329 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.048309    6931 scope.go:122] "RemoveContainer" containerID="6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"
Oct 31 04:53:02.048631 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:53:02.048608    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2\": container with ID starting with 6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2 not found: ID does not exist" containerID="6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"
Oct 31 04:53:02.048705 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.048642    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2"} err="failed to get container status \"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2\": rpc error: code = NotFound desc = could not find container \"6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2\": container with ID starting with 6ded151c6aa35651923af63232b66325379ec7766610941999f2c1c2cea775b2 not found: ID does not exist"
Oct 31 04:53:02.084704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.084670    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:53:02.096700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.096657    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6c5mqc6"]
Oct 31 04:53:02.123573 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.123545    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ed23c3-badb-455a-b982-2309ca2adbfb" path="/var/lib/kubelet/pods/04ed23c3-badb-455a-b982-2309ca2adbfb/volumes"
Oct 31 04:53:02.581711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:02.581674    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:04.385971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:04.385938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:06.131776 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:06.131732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:08.012288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:08.012243    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:09.127123 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:09.127083    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:09.747542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:09.747497    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:10.798797 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:53:10.798754    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:53:11.908489 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:11.908445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:14.009004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:14.008963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:16.047863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:16.047824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:18.001817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:18.001780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:19.877472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:19.877432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:21.619296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:21.619251    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:23.429534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:23.429494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:25.234381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:25.234346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:27.007236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:27.007198    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:28.904289 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:28.904253    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:29.108605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:29.108568    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:53:29.228590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:29.228493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:29.471133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:29.459682    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:31.001506 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:31.001458    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:32.800300 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:32.800224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:32.825365 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:32.825326    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:53:33.012014 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:33.011977    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:33.228094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:33.228000    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:34.109503 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.109442    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" podUID="1734993c-a11c-407e-8e4b-4fc5fba766f0" containerName="payload-processing" containerID="cri-o://ff68efa55599d19a7546128fd903158ddb6ca35c6602ec6972a645d03d912723" gracePeriod=25
Oct 31 04:53:34.328842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.328809    6931 generic.go:356] "Generic (PLEG): container finished" podID="1734993c-a11c-407e-8e4b-4fc5fba766f0" containerID="ff68efa55599d19a7546128fd903158ddb6ca35c6602ec6972a645d03d912723" exitCode=0
Oct 31 04:53:34.329047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.328889    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" event={"ID":"1734993c-a11c-407e-8e4b-4fc5fba766f0","Type":"ContainerDied","Data":"ff68efa55599d19a7546128fd903158ddb6ca35c6602ec6972a645d03d912723"}
Oct 31 04:53:34.383608 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.383580    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:53:34.428035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.427994    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\") pod \"1734993c-a11c-407e-8e4b-4fc5fba766f0\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") "
Oct 31 04:53:34.428260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428060    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\") pod \"1734993c-a11c-407e-8e4b-4fc5fba766f0\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") "
Oct 31 04:53:34.428260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428092    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\") pod \"1734993c-a11c-407e-8e4b-4fc5fba766f0\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") "
Oct 31 04:53:34.428260 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428151    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\") pod \"1734993c-a11c-407e-8e4b-4fc5fba766f0\" (UID: \"1734993c-a11c-407e-8e4b-4fc5fba766f0\") "
Oct 31 04:53:34.428691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428558    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume" pod "1734993c-a11c-407e-8e4b-4fc5fba766f0" (UID: "1734993c-a11c-407e-8e4b-4fc5fba766f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:34.428691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428641    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume" pod "1734993c-a11c-407e-8e4b-4fc5fba766f0" (UID: "1734993c-a11c-407e-8e4b-4fc5fba766f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:34.428691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.428664    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:34.430708 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.430681    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9" pod "1734993c-a11c-407e-8e4b-4fc5fba766f0" (UID: "1734993c-a11c-407e-8e4b-4fc5fba766f0"). InnerVolumeSpecName "kube-api-access-7tbn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:34.430859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.430743    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials" pod "1734993c-a11c-407e-8e4b-4fc5fba766f0" (UID: "1734993c-a11c-407e-8e4b-4fc5fba766f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:34.529472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.529419    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1734993c-a11c-407e-8e4b-4fc5fba766f0-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:34.529472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.529469    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:34.529472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.529481    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbn9\" (UniqueName: \"kubernetes.io/projected/1734993c-a11c-407e-8e4b-4fc5fba766f0-kube-api-access-7tbn9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:34.778668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.778583    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:34.992458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:34.992417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:35.385181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.384670    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:35.392211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.390265    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q" event={"ID":"1734993c-a11c-407e-8e4b-4fc5fba766f0","Type":"ContainerDied","Data":"4f7349f6e6dd8c11c4746359cfcbfc91422108545bf2ac6da9ef9b8e30dce9d9"}
Oct 31 04:53:35.392211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.390350    6931 scope.go:122] "RemoveContainer" containerID="ff68efa55599d19a7546128fd903158ddb6ca35c6602ec6972a645d03d912723"
Oct 31 04:53:35.392211 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.390522    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"
Oct 31 04:53:35.487908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.487836    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:53:35.515180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.502606    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-7fc8kh55q"]
Oct 31 04:53:35.601743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:35.601695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:36.124862 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:36.124829    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1734993c-a11c-407e-8e4b-4fc5fba766f0" path="/var/lib/kubelet/pods/1734993c-a11c-407e-8e4b-4fc5fba766f0/volumes"
Oct 31 04:53:36.709398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:36.709362    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:37.826426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:37.826374    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" podUID="a6201d40-fdfe-45be-9075-65eb88e05139" containerName="payload-processing" containerID="cri-o://8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034" gracePeriod=25
Oct 31 04:53:38.103469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.103439    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:53:38.184060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.184021    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\") pod \"a6201d40-fdfe-45be-9075-65eb88e05139\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") "
Oct 31 04:53:38.184248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.184228    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\") pod \"a6201d40-fdfe-45be-9075-65eb88e05139\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") "
Oct 31 04:53:38.184320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.184298    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\") pod \"a6201d40-fdfe-45be-9075-65eb88e05139\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") "
Oct 31 04:53:38.184378 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.184362    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\") pod \"a6201d40-fdfe-45be-9075-65eb88e05139\" (UID: \"a6201d40-fdfe-45be-9075-65eb88e05139\") "
Oct 31 04:53:38.185453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.184876    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume" pod "a6201d40-fdfe-45be-9075-65eb88e05139" (UID: "a6201d40-fdfe-45be-9075-65eb88e05139"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:38.185453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.185237    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:38.185769 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.185739    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume" pod "a6201d40-fdfe-45be-9075-65eb88e05139" (UID: "a6201d40-fdfe-45be-9075-65eb88e05139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:38.186661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.186638    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw" pod "a6201d40-fdfe-45be-9075-65eb88e05139" (UID: "a6201d40-fdfe-45be-9075-65eb88e05139"). InnerVolumeSpecName "kube-api-access-skdbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:38.186980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.186966    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials" pod "a6201d40-fdfe-45be-9075-65eb88e05139" (UID: "a6201d40-fdfe-45be-9075-65eb88e05139"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:38.286485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.286452    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6201d40-fdfe-45be-9075-65eb88e05139-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:38.286485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.286478    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:38.286485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.286488    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdbw\" (UniqueName: \"kubernetes.io/projected/a6201d40-fdfe-45be-9075-65eb88e05139-kube-api-access-skdbw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:38.412140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.412051    6931 generic.go:356] "Generic (PLEG): container finished" podID="a6201d40-fdfe-45be-9075-65eb88e05139" containerID="8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034" exitCode=0
Oct 31 04:53:38.412140 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.412101    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"
Oct 31 04:53:38.412421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.412131    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" event={"ID":"a6201d40-fdfe-45be-9075-65eb88e05139","Type":"ContainerDied","Data":"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"}
Oct 31 04:53:38.412421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.412217    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96" event={"ID":"a6201d40-fdfe-45be-9075-65eb88e05139","Type":"ContainerDied","Data":"06975d1fa96c0907172589c0a72b89c37333ef6685956625a5c1129ec376f4a5"}
Oct 31 04:53:38.412421 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.412247    6931 scope.go:122] "RemoveContainer" containerID="8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"
Oct 31 04:53:38.423596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.423575    6931 scope.go:122] "RemoveContainer" containerID="8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"
Oct 31 04:53:38.423915 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:53:38.423896    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034\": container with ID starting with 8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034 not found: ID does not exist" containerID="8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"
Oct 31 04:53:38.423973 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.423923    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034"} err="failed to get container status \"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034\": rpc error: code = NotFound desc = could not find container \"8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034\": container with ID starting with 8c82dc3e4861fff67b41ed57abffabad494a72ae91997383c4f27c1bc7e4f034 not found: ID does not exist"
Oct 31 04:53:38.463474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.463425    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:53:38.470779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.470753    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf99f468c-xjg96"]
Oct 31 04:53:38.625239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:38.625205    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:39.326463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:39.326421    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:39.543442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:39.543405    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:40.126264 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:40.126224    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6201d40-fdfe-45be-9075-65eb88e05139" path="/var/lib/kubelet/pods/a6201d40-fdfe-45be-9075-65eb88e05139/volumes"
Oct 31 04:53:40.514799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:40.514703    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:40.758976 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:53:40.758939    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:53:41.113880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.113847    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:41.172145 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172109    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 04:53:41.172759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172744    6931 cpu_manager.go:414] "containerMap: removing container" podUID="a6201d40-fdfe-45be-9075-65eb88e05139" containerName="payload-processing"
Oct 31 04:53:41.172806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172760    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6201d40-fdfe-45be-9075-65eb88e05139" containerName="payload-processing"
Oct 31 04:53:41.172806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172800    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1734993c-a11c-407e-8e4b-4fc5fba766f0" containerName="payload-processing"
Oct 31 04:53:41.172894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172806    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1734993c-a11c-407e-8e4b-4fc5fba766f0" containerName="payload-processing"
Oct 31 04:53:41.172894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172830    6931 cpu_manager.go:414] "containerMap: removing container" podUID="04ed23c3-badb-455a-b982-2309ca2adbfb" containerName="payload-processing"
Oct 31 04:53:41.172894 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172835    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ed23c3-badb-455a-b982-2309ca2adbfb" containerName="payload-processing"
Oct 31 04:53:41.173001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172915    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ed23c3-badb-455a-b982-2309ca2adbfb" containerName="payload-processing"
Oct 31 04:53:41.173001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172923    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6201d40-fdfe-45be-9075-65eb88e05139" containerName="payload-processing"
Oct 31 04:53:41.173001 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.172936    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1734993c-a11c-407e-8e4b-4fc5fba766f0" containerName="payload-processing"
Oct 31 04:53:41.177677 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.177655    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.196242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.196211    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 04:53:41.205522 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.205482    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-670795-dockercfg-fcp4r\""
Oct 31 04:53:41.205723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.205585    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-670795\""
Oct 31 04:53:41.224381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.224338    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.224595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.224482    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwj8\" (UniqueName: \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.304062 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.304024    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 04:53:41.311605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.311572    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.312068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.312046    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 04:53:41.330557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.329885    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.330557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.329999    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwj8\" (UniqueName: \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.331095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.331053    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.343291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.343263    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:41.353485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.353457    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwj8\" (UniqueName: \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\") pod \"payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.468241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.467192    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.468241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.467250    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87g7z\" (UniqueName: \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.493177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.492383    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:41.541214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.534622    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:53:41.575487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.575445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:41.615294 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.615257    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.619895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.619842    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87g7z\" (UniqueName: \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.620505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.620476    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.646967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.646923    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87g7z\" (UniqueName: \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\") pod \"payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:41.793617 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.793576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:41.890509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.890480    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 04:53:41.894304 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:53:41.894254    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e60e01_bce0_4ea9_aad3_f888dda53189.slice/crio-5d24df986aa8f46e802b59b378deaebac6030fe41a516f64dc9655db7a38a677 WatchSource:0}: Error finding container 5d24df986aa8f46e802b59b378deaebac6030fe41a516f64dc9655db7a38a677: Status 404 returned error can't find the container with id 5d24df986aa8f46e802b59b378deaebac6030fe41a516f64dc9655db7a38a677
Oct 31 04:53:41.939967 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:41.939652    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:42.045970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.045784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:42.325414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.325387    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 04:53:42.328115 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:53:42.328094    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c539ae_d39a_42c8_b1e3_2754023442fc.slice/crio-fbba12a0f2a30c5b269190ab74c983a93d91328688325dae2fcad880e24c9ee4 WatchSource:0}: Error finding container fbba12a0f2a30c5b269190ab74c983a93d91328688325dae2fcad880e24c9ee4: Status 404 returned error can't find the container with id fbba12a0f2a30c5b269190ab74c983a93d91328688325dae2fcad880e24c9ee4
Oct 31 04:53:42.476102 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.476064    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" event={"ID":"23c539ae-d39a-42c8-b1e3-2754023442fc","Type":"ContainerStarted","Data":"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"}
Oct 31 04:53:42.476296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.476133    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:42.476296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.476171    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" event={"ID":"23c539ae-d39a-42c8-b1e3-2754023442fc","Type":"ContainerStarted","Data":"fbba12a0f2a30c5b269190ab74c983a93d91328688325dae2fcad880e24c9ee4"}
Oct 31 04:53:42.477608 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.477585    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" event={"ID":"d7e60e01-bce0-4ea9-aad3-f888dda53189","Type":"ContainerStarted","Data":"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"}
Oct 31 04:53:42.477730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.477627    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" event={"ID":"d7e60e01-bce0-4ea9-aad3-f888dda53189","Type":"ContainerStarted","Data":"5d24df986aa8f46e802b59b378deaebac6030fe41a516f64dc9655db7a38a677"}
Oct 31 04:53:42.477730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.477669    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:42.516616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.516564    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" podStartSLOduration=1.516550987 podStartE2EDuration="1.516550987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:42.515647117 +0000 UTC m=+2918.548646812" watchObservedRunningTime="2026-10-31 04:53:42.516550987 +0000 UTC m=+2918.549550666"
Oct 31 04:53:42.556125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.556070    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" podStartSLOduration=1.5560556079999999 podStartE2EDuration="1.556055608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:42.554530962 +0000 UTC m=+2918.587530641" watchObservedRunningTime="2026-10-31 04:53:42.556055608 +0000 UTC m=+2918.589055287"
Oct 31 04:53:42.802397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:42.802356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:44.601590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:44.601552    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:44.863704 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:44.863621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:46.465271 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.465234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:46.535634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.535587    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" podUID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" containerName="payload-processing" containerID="cri-o://5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20" gracePeriod=25
Oct 31 04:53:46.804480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.804451    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:53:46.903317 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.903280    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\") pod \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") "
Oct 31 04:53:46.903513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.903416    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\") pod \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") "
Oct 31 04:53:46.903584 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.903569    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\") pod \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") "
Oct 31 04:53:46.903645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.903622    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\") pod \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\" (UID: \"4879ed6b-8a5f-4b15-ad5c-f3b959eea887\") "
Oct 31 04:53:46.903717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.903689    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume" pod "4879ed6b-8a5f-4b15-ad5c-f3b959eea887" (UID: "4879ed6b-8a5f-4b15-ad5c-f3b959eea887"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:46.904065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.904036    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume" pod "4879ed6b-8a5f-4b15-ad5c-f3b959eea887" (UID: "4879ed6b-8a5f-4b15-ad5c-f3b959eea887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:53:46.904207 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.904094    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:46.905559 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.905536    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials" pod "4879ed6b-8a5f-4b15-ad5c-f3b959eea887" (UID: "4879ed6b-8a5f-4b15-ad5c-f3b959eea887"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:46.905641 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:46.905624    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk" pod "4879ed6b-8a5f-4b15-ad5c-f3b959eea887" (UID: "4879ed6b-8a5f-4b15-ad5c-f3b959eea887"). InnerVolumeSpecName "kube-api-access-k2pzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:53:47.005245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.005207    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pzk\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-kube-api-access-k2pzk\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:47.005245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.005239    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:47.005245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.005249    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4879ed6b-8a5f-4b15-ad5c-f3b959eea887-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:53:47.510088 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.510050    6931 generic.go:356] "Generic (PLEG): container finished" podID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" containerID="5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20" exitCode=0
Oct 31 04:53:47.510595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.510101    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"
Oct 31 04:53:47.510595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.510131    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" event={"ID":"4879ed6b-8a5f-4b15-ad5c-f3b959eea887","Type":"ContainerDied","Data":"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"}
Oct 31 04:53:47.510595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.510220    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg" event={"ID":"4879ed6b-8a5f-4b15-ad5c-f3b959eea887","Type":"ContainerDied","Data":"cfb906e65e7fb574ccb274e66c54ef136628711e70153e50c075929ea8b34d10"}
Oct 31 04:53:47.510595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.510244    6931 scope.go:122] "RemoveContainer" containerID="5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"
Oct 31 04:53:47.522681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.522657    6931 scope.go:122] "RemoveContainer" containerID="5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"
Oct 31 04:53:47.523016 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:53:47.522993    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20\": container with ID starting with 5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20 not found: ID does not exist" containerID="5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"
Oct 31 04:53:47.523087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.523027    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20"} err="failed to get container status \"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20\": rpc error: code = NotFound desc = could not find container \"5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20\": container with ID starting with 5612b96205907dda6e5b20291ea2a396fce7754dfc97721bf296051406ad8d20 not found: ID does not exist"
Oct 31 04:53:47.560458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.560425    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:53:47.571899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:47.571871    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-6bbfm95rg"]
Oct 31 04:53:48.124023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:48.123992    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" path="/var/lib/kubelet/pods/4879ed6b-8a5f-4b15-ad5c-f3b959eea887/volumes"
Oct 31 04:53:48.221177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:48.218620    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:48.439934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:48.439860    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:48.659150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:48.659113    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:48.985040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:48.985004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:49.658732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:49.658692    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:49.885994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:49.885954    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:50.114932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:50.106771    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:51.975180 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:51.971881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:52.124885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.124855    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:53:52.125487 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.125472    6931 cpu_manager.go:414] "containerMap: removing container" podUID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" containerName="payload-processing"
Oct 31 04:53:52.125534 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.125488    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" containerName="payload-processing"
Oct 31 04:53:52.125590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.125580    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4879ed6b-8a5f-4b15-ad5c-f3b959eea887" containerName="payload-processing"
Oct 31 04:53:52.130497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.130477    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.145945 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.145916    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:53:52.154830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.154803    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"openshift-service-ca.crt\""
Oct 31 04:53:52.155051 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.154841    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"routing-overlay\""
Oct 31 04:53:52.155187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.155034    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"kube-root-ca.crt\""
Oct 31 04:53:52.156689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.156670    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"agc-cred-e1685c734038\""
Oct 31 04:53:52.156786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.156679    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8973e2\""
Oct 31 04:53:52.156786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.156728    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8973e2\"/\"payload-processing-external-model-e2e--508e01fb-dockercfg-h6ghp\""
Oct 31 04:53:52.294475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.294435    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.294747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.294726    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsbz\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.313484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.313438    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.314184 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.314136    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.416108    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.416208    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsbz\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.416349    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.416417    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418219 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.417438    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.418994 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.418967    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.429136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.426831    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.440898 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.440866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:52.441948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.441921    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsbz\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\") pod \"payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") " pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.444817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.444787    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:52.753796 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:52.753770    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:53:52.756810 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:53:52.756779    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf844f32_3c41_4ac9_997b_918138264172.slice/crio-01394cf87d45d6a815ae6930ae2ca184d3ec2607c743eedd9985c812fa3fd9ca WatchSource:0}: Error finding container 01394cf87d45d6a815ae6930ae2ca184d3ec2607c743eedd9985c812fa3fd9ca: Status 404 returned error can't find the container with id 01394cf87d45d6a815ae6930ae2ca184d3ec2607c743eedd9985c812fa3fd9ca
Oct 31 04:53:53.486903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.486872    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 04:53:53.487433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.486941    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 04:53:53.555773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.555732    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" event={"ID":"df844f32-3c41-4ac9-997b-918138264172","Type":"ContainerStarted","Data":"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"}
Oct 31 04:53:53.555976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.555795    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" event={"ID":"df844f32-3c41-4ac9-997b-918138264172","Type":"ContainerStarted","Data":"01394cf87d45d6a815ae6930ae2ca184d3ec2607c743eedd9985c812fa3fd9ca"}
Oct 31 04:53:53.555976 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.555817    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:53.617842 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.617780    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" podStartSLOduration=1.617760726 podStartE2EDuration="1.617760726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:53.614588004 +0000 UTC m=+2929.647587719" watchObservedRunningTime="2026-10-31 04:53:53.617760726 +0000 UTC m=+2929.650760406"
Oct 31 04:53:53.879689 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:53.879604    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:54.018031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.017998    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:53:54.022444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.022421    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.038230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.038198    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-e2e--299d87ac-dockercfg-j9bmb\""
Oct 31 04:53:54.038230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.038228    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"agc-cred-b4af2923f59b\""
Oct 31 04:53:54.055020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.054972    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:53:54.059314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.059271    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"kube-root-ca.crt\""
Oct 31 04:53:54.059314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.059272    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"payload-processing-external-model-plugins-e2e-extmod-b-227c93\""
Oct 31 04:53:54.059505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.059375    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"openshift-service-ca.crt\""
Oct 31 04:53:54.059997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.059968    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-227c93\"/\"routing-overlay\""
Oct 31 04:53:54.139663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.139578    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.139663 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.139660    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjpx\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.139858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.139785    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.139858 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.139830    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.244060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.243970    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjpx\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.244427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.244406    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.244582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.244567    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.244780 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.244761    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.256360 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.246825    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.256853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.256821    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.257805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.257744    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.274149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.274113    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjpx\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\") pod \"payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") " pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.338473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.338434    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:54.532542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.532506    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:54.649820 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:54.649720    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:53:54.660456 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:53:54.660422    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc083c62_e88e_4bb9_897c_96929173beb3.slice/crio-3f56e0d3fee3c6275b3c3c4780ab285f5e6f9744b7143f1e80d339bd128eff3b WatchSource:0}: Error finding container 3f56e0d3fee3c6275b3c3c4780ab285f5e6f9744b7143f1e80d339bd128eff3b: Status 404 returned error can't find the container with id 3f56e0d3fee3c6275b3c3c4780ab285f5e6f9744b7143f1e80d339bd128eff3b
Oct 31 04:53:55.569883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:55.569844    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" event={"ID":"bc083c62-e88e-4bb9-897c-96929173beb3","Type":"ContainerStarted","Data":"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"}
Oct 31 04:53:55.570361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:55.569899    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" event={"ID":"bc083c62-e88e-4bb9-897c-96929173beb3","Type":"ContainerStarted","Data":"3f56e0d3fee3c6275b3c3c4780ab285f5e6f9744b7143f1e80d339bd128eff3b"}
Oct 31 04:53:55.570361 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:55.569970    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:53:55.614182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:55.614104    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" podStartSLOduration=2.6140868189999997 podStartE2EDuration="2.614086819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:53:55.610998872 +0000 UTC m=+2931.643998587" watchObservedRunningTime="2026-10-31 04:53:55.614086819 +0000 UTC m=+2931.647086489"
Oct 31 04:53:56.381869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:56.381836    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:57.103825 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:57.103788    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:53:57.108032 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:57.108003    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:53:57.278981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:57.278947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:57.502234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:57.501266    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:57.912125 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:57.912094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:58.037234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:58.037197    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:53:58.216808 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:58.216720    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:58.505629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:53:58.505526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:00.411859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:00.411814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:01.558023 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:01.557980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:02.104892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.104839    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" podUID="df844f32-3c41-4ac9-997b-918138264172" containerName="payload-processing" containerID="cri-o://c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954" gracePeriod=25
Oct 31 04:54:02.210096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.210035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:02.371638 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.371575    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:54:02.469525 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.469488    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\") pod \"df844f32-3c41-4ac9-997b-918138264172\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") "
Oct 31 04:54:02.469701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.469621    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\") pod \"df844f32-3c41-4ac9-997b-918138264172\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") "
Oct 31 04:54:02.469701 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.469697    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\") pod \"df844f32-3c41-4ac9-997b-918138264172\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") "
Oct 31 04:54:02.469822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.469749    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\") pod \"df844f32-3c41-4ac9-997b-918138264172\" (UID: \"df844f32-3c41-4ac9-997b-918138264172\") "
Oct 31 04:54:02.470068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.470036    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume" pod "df844f32-3c41-4ac9-997b-918138264172" (UID: "df844f32-3c41-4ac9-997b-918138264172"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:02.470188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.470085    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume" pod "df844f32-3c41-4ac9-997b-918138264172" (UID: "df844f32-3c41-4ac9-997b-918138264172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:02.470296 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.470278    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:02.470335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.470304    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df844f32-3c41-4ac9-997b-918138264172-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:02.471831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.471812    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials" pod "df844f32-3c41-4ac9-997b-918138264172" (UID: "df844f32-3c41-4ac9-997b-918138264172"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:02.471907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.471879    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz" pod "df844f32-3c41-4ac9-997b-918138264172" (UID: "df844f32-3c41-4ac9-997b-918138264172"). InnerVolumeSpecName "kube-api-access-wcsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:02.571784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.571742    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsbz\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-kube-api-access-wcsbz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:02.571784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.571776    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df844f32-3c41-4ac9-997b-918138264172-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:02.614538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.614496    6931 generic.go:356] "Generic (PLEG): container finished" podID="df844f32-3c41-4ac9-997b-918138264172" containerID="c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954" exitCode=0
Oct 31 04:54:02.614745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.614543    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"
Oct 31 04:54:02.614745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.614576    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" event={"ID":"df844f32-3c41-4ac9-997b-918138264172","Type":"ContainerDied","Data":"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"}
Oct 31 04:54:02.614745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.614627    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb" event={"ID":"df844f32-3c41-4ac9-997b-918138264172","Type":"ContainerDied","Data":"01394cf87d45d6a815ae6930ae2ca184d3ec2607c743eedd9985c812fa3fd9ca"}
Oct 31 04:54:02.614745 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.614649    6931 scope.go:122] "RemoveContainer" containerID="c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"
Oct 31 04:54:02.627821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.627800    6931 scope.go:122] "RemoveContainer" containerID="c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"
Oct 31 04:54:02.628185 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:02.628149    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954\": container with ID starting with c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954 not found: ID does not exist" containerID="c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"
Oct 31 04:54:02.628263 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.628196    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954"} err="failed to get container status \"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954\": rpc error: code = NotFound desc = could not find container \"c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954\": container with ID starting with c69234480eaa3f58ad6864f695713108e133604aadf4b5ca30f7adb8ae4ab954 not found: ID does not exist"
Oct 31 04:54:02.665510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.665476    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:54:02.669038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:02.669009    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8973e2/payload-processing-external-model-e2e-extmod-a-8973e2-d86dffxgb"]
Oct 31 04:54:03.038399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.038349    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" podUID="bc083c62-e88e-4bb9-897c-96929173beb3" containerName="payload-processing" containerID="cri-o://7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93" gracePeriod=25
Oct 31 04:54:03.324781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.324749    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:54:03.382049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382018    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\") pod \"bc083c62-e88e-4bb9-897c-96929173beb3\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") "
Oct 31 04:54:03.382239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382227    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\") pod \"bc083c62-e88e-4bb9-897c-96929173beb3\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") "
Oct 31 04:54:03.382330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382307    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\") pod \"bc083c62-e88e-4bb9-897c-96929173beb3\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") "
Oct 31 04:54:03.382407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382338    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\") pod \"bc083c62-e88e-4bb9-897c-96929173beb3\" (UID: \"bc083c62-e88e-4bb9-897c-96929173beb3\") "
Oct 31 04:54:03.382653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382480    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume" pod "bc083c62-e88e-4bb9-897c-96929173beb3" (UID: "bc083c62-e88e-4bb9-897c-96929173beb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:03.382756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.382723    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume" pod "bc083c62-e88e-4bb9-897c-96929173beb3" (UID: "bc083c62-e88e-4bb9-897c-96929173beb3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:03.383249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.383223    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:03.383349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.383253    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc083c62-e88e-4bb9-897c-96929173beb3-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:03.384529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.384509    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials" pod "bc083c62-e88e-4bb9-897c-96929173beb3" (UID: "bc083c62-e88e-4bb9-897c-96929173beb3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:03.384632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.384538    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx" pod "bc083c62-e88e-4bb9-897c-96929173beb3" (UID: "bc083c62-e88e-4bb9-897c-96929173beb3"). InnerVolumeSpecName "kube-api-access-ssjpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:03.485011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.484971    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:03.485011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.485010    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjpx\" (UniqueName: \"kubernetes.io/projected/bc083c62-e88e-4bb9-897c-96929173beb3-kube-api-access-ssjpx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:03.622709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.622623    6931 generic.go:356] "Generic (PLEG): container finished" podID="bc083c62-e88e-4bb9-897c-96929173beb3" containerID="7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93" exitCode=0
Oct 31 04:54:03.622709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.622668    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"
Oct 31 04:54:03.623301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.622719    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" event={"ID":"bc083c62-e88e-4bb9-897c-96929173beb3","Type":"ContainerDied","Data":"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"}
Oct 31 04:54:03.623301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.622762    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b" event={"ID":"bc083c62-e88e-4bb9-897c-96929173beb3","Type":"ContainerDied","Data":"3f56e0d3fee3c6275b3c3c4780ab285f5e6f9744b7143f1e80d339bd128eff3b"}
Oct 31 04:54:03.623301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.622788    6931 scope.go:122] "RemoveContainer" containerID="7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"
Oct 31 04:54:03.633651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.633631    6931 scope.go:122] "RemoveContainer" containerID="7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"
Oct 31 04:54:03.633928 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:03.633910    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93\": container with ID starting with 7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93 not found: ID does not exist" containerID="7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"
Oct 31 04:54:03.633993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.633935    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93"} err="failed to get container status \"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93\": rpc error: code = NotFound desc = could not find container \"7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93\": container with ID starting with 7ac59caf272073163b0c8f6e22d4476ddb73383c5a4f6f5601fa70be72836a93 not found: ID does not exist"
Oct 31 04:54:03.674351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.674314    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:54:03.686083 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.686048    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-227c93/payload-processing-external-model-e2e-extmod-b-227c93-64bc87l6b"]
Oct 31 04:54:03.998237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:03.993472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:04.125042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:04.125010    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc083c62-e88e-4bb9-897c-96929173beb3" path="/var/lib/kubelet/pods/bc083c62-e88e-4bb9-897c-96929173beb3/volumes"
Oct 31 04:54:04.125480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:04.125464    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df844f32-3c41-4ac9-997b-918138264172" path="/var/lib/kubelet/pods/df844f32-3c41-4ac9-997b-918138264172/volumes"
Oct 31 04:54:05.791415 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:05.791369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:06.016613 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:06.016578    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:07.814982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:07.814939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:09.271658 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:09.271615    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:09.482975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:09.482937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:09.984576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:09.984535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:10.760396 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:10.760353    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:54:11.001737 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.001693    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:11.016275 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016192    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 04:54:11.016837 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016822    6931 cpu_manager.go:414] "containerMap: removing container" podUID="df844f32-3c41-4ac9-997b-918138264172" containerName="payload-processing"
Oct 31 04:54:11.016884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016839    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df844f32-3c41-4ac9-997b-918138264172" containerName="payload-processing"
Oct 31 04:54:11.016884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016876    6931 cpu_manager.go:414] "containerMap: removing container" podUID="bc083c62-e88e-4bb9-897c-96929173beb3" containerName="payload-processing"
Oct 31 04:54:11.016884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016881    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc083c62-e88e-4bb9-897c-96929173beb3" containerName="payload-processing"
Oct 31 04:54:11.016990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016981    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc083c62-e88e-4bb9-897c-96929173beb3" containerName="payload-processing"
Oct 31 04:54:11.017025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.016995    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="df844f32-3c41-4ac9-997b-918138264172" containerName="payload-processing"
Oct 31 04:54:11.021974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.021958    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.038970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.038943    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 04:54:11.047707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.047681    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-171edd\""
Oct 31 04:54:11.047707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.047683    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-171edd-dockercfg-4dpgf\""
Oct 31 04:54:11.090087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.090052    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 04:54:11.094299 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.094279    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.108290 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.108256    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxnq\" (UniqueName: \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.108431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.108304    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.108482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.108461    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wspq\" (UniqueName: \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.108585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.108566    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.115344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.115299    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 04:54:11.241971 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.213884    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wspq\" (UniqueName: \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.278645 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.273173    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wspq\" (UniqueName: \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.279519 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.279488    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.280435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.280412    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxnq\" (UniqueName: \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.280926 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.280904    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.281683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.280576    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.283171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.283132    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-171edd-766c966749-c55nm\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") " pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.286872 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.286849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:11.304553 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.304520    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxnq\" (UniqueName: \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\") pod \"payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.335241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.335213    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:11.411249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.409938    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:11.642928 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.642905    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 04:54:11.645742 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:11.645714    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc6db11_c632_4c17_93b1_ac9c8e2cc246.slice/crio-41eb9d2b43d3acf158a589bf165618524c3807dfb54c1e135b85d15421faa903 WatchSource:0}: Error finding container 41eb9d2b43d3acf158a589bf165618524c3807dfb54c1e135b85d15421faa903: Status 404 returned error can't find the container with id 41eb9d2b43d3acf158a589bf165618524c3807dfb54c1e135b85d15421faa903
Oct 31 04:54:11.704715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.704620    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" event={"ID":"1fc6db11-c632-4c17-93b1-ac9c8e2cc246","Type":"ContainerStarted","Data":"41eb9d2b43d3acf158a589bf165618524c3807dfb54c1e135b85d15421faa903"}
Oct 31 04:54:11.745590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.745526    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 04:54:11.750462 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:11.750425    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f11c6a_23af_4678_932c_1aa7c4b85242.slice/crio-5ca2b25bd7e99afa844f8d1c38d148118d76f397b54349cb4b3edce127f8b69b WatchSource:0}: Error finding container 5ca2b25bd7e99afa844f8d1c38d148118d76f397b54349cb4b3edce127f8b69b: Status 404 returned error can't find the container with id 5ca2b25bd7e99afa844f8d1c38d148118d76f397b54349cb4b3edce127f8b69b
Oct 31 04:54:11.772609 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:11.772577    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:12.713696 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.713661    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" event={"ID":"1fc6db11-c632-4c17-93b1-ac9c8e2cc246","Type":"ContainerStarted","Data":"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"}
Oct 31 04:54:12.713980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.713749    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:12.715442 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.715418    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" event={"ID":"56f11c6a-23af-4678-932c-1aa7c4b85242","Type":"ContainerStarted","Data":"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"}
Oct 31 04:54:12.715547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.715457    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" event={"ID":"56f11c6a-23af-4678-932c-1aa7c4b85242","Type":"ContainerStarted","Data":"5ca2b25bd7e99afa844f8d1c38d148118d76f397b54349cb4b3edce127f8b69b"}
Oct 31 04:54:12.715547 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.715484    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:12.753740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.753685    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" podStartSLOduration=2.753671681 podStartE2EDuration="2.753671681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:12.752887507 +0000 UTC m=+2948.785887190" watchObservedRunningTime="2026-10-31 04:54:12.753671681 +0000 UTC m=+2948.786671354"
Oct 31 04:54:12.794846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:12.794788    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" podStartSLOduration=1.794772477 podStartE2EDuration="1.794772477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:12.792443807 +0000 UTC m=+2948.825443487" watchObservedRunningTime="2026-10-31 04:54:12.794772477 +0000 UTC m=+2948.827772157"
Oct 31 04:54:13.559759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:13.559721    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:16.034182 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:16.024966    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:18.126315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:18.126273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:20.024109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:20.024070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:22.012085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:22.012043    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:22.687320 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:22.687278    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:22.901097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:22.901053    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:23.728670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:23.728633    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 04:54:23.729959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:23.728731    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 04:54:24.401335 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.401291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:24.544542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.544504    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:24.550340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.550316    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.582406 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582371    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-1fdb79c63432\""
Oct 31 04:54:24.582666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582470    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ded435\""
Oct 31 04:54:24.582666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582514    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"openshift-service-ca.crt\""
Oct 31 04:54:24.582666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582606    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"kube-root-ca.crt\""
Oct 31 04:54:24.582871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582770    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"routing-overlay\""
Oct 31 04:54:24.582871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582843    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"agc-cred-425c31a80fac\""
Oct 31 04:54:24.583004 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.582974    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ded435\"/\"payload-processing-external-model-e2e--b0123f89-dockercfg-54866\""
Oct 31 04:54:24.586273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.586247    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:24.624635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.624601    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.624817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.624769    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.624865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.624821    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.624958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.624941    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn89m\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.726230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726114    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.726230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726223    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn89m\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.726436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726279    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.726436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726379    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.726779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726757    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.727003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.726979    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.728806 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.728784    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.748781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.748755    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn89m\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\") pod \"payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") " pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:24.866052 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:24.866004    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:25.171232 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.171199    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:25.174104 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:25.174078    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df4d7ed_7d34_4a08_965c_7e84a3ce36cd.slice/crio-40c8189a9b61044791b00c0c35619244800f9a0a0cb91084f907c360c1bb92c8 WatchSource:0}: Error finding container 40c8189a9b61044791b00c0c35619244800f9a0a0cb91084f907c360c1bb92c8: Status 404 returned error can't find the container with id 40c8189a9b61044791b00c0c35619244800f9a0a0cb91084f907c360c1bb92c8
Oct 31 04:54:25.334060 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.334020    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:25.338468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.338444    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.354678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.354526    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"agc-cred-1ee228dee901\""
Oct 31 04:54:25.354678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.354549    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-plugins-e2e-extmod-a-654741\""
Oct 31 04:54:25.354678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.354606    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"openshift-service-ca.crt\""
Oct 31 04:54:25.354678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.354612    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"routing-overlay\""
Oct 31 04:54:25.354678 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.354610    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"payload-processing-external-model-e2e--dfe12206-dockercfg-jwsgv\""
Oct 31 04:54:25.371206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.366354    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"agc-cred-8c9201ffc741\""
Oct 31 04:54:25.371206 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.366367    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-654741\"/\"kube-root-ca.crt\""
Oct 31 04:54:25.371399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.371318    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:25.435535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.435446    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.435686 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.435598    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.435730 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.435712    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.435773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.435759    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bwp\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.537467 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.537427    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.537675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.537491    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bwp\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.537675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.537606    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.537805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.537707    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.538354 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.538322    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.538458 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.538422    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.540024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.540001    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.562767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.562740    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bwp\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\") pod \"payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") " pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.652694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.652653    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:25.863074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.863033    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" event={"ID":"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd","Type":"ContainerStarted","Data":"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"}
Oct 31 04:54:25.863482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.863088    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" event={"ID":"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd","Type":"ContainerStarted","Data":"40c8189a9b61044791b00c0c35619244800f9a0a0cb91084f907c360c1bb92c8"}
Oct 31 04:54:25.863482 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.863113    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:25.905504 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.905444    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" podStartSLOduration=1.905430253 podStartE2EDuration="1.905430253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:25.90299617 +0000 UTC m=+2961.935995848" watchObservedRunningTime="2026-10-31 04:54:25.905430253 +0000 UTC m=+2961.938429931"
Oct 31 04:54:25.959815 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:25.959379    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:25.962723 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:25.962701    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b274d9_5d25_4298_94dc_80758a2ff93c.slice/crio-6076dda3cd796a0266fb4b9855b2f66ca52969bd563c7dba609dbbba1d26074c WatchSource:0}: Error finding container 6076dda3cd796a0266fb4b9855b2f66ca52969bd563c7dba609dbbba1d26074c: Status 404 returned error can't find the container with id 6076dda3cd796a0266fb4b9855b2f66ca52969bd563c7dba609dbbba1d26074c
Oct 31 04:54:26.199555 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:26.199516    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:26.871574 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:26.871535    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" event={"ID":"d2b274d9-5d25-4298-94dc-80758a2ff93c","Type":"ContainerStarted","Data":"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"}
Oct 31 04:54:26.872078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:26.871600    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" event={"ID":"d2b274d9-5d25-4298-94dc-80758a2ff93c","Type":"ContainerStarted","Data":"6076dda3cd796a0266fb4b9855b2f66ca52969bd563c7dba609dbbba1d26074c"}
Oct 31 04:54:26.872078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:26.871758    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:26.914699 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:26.914637    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" podStartSLOduration=1.914622156 podStartE2EDuration="1.914622156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:26.911282148 +0000 UTC m=+2962.944281827" watchObservedRunningTime="2026-10-31 04:54:26.914622156 +0000 UTC m=+2962.947621832"
Oct 31 04:54:28.001147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:28.001108    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:29.574795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.574755    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 04:54:29.579977 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.579952    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.587480 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.587449    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:54:29.587647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.587624    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" containerID="cri-o://d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413" gracePeriod=30
Oct 31 04:54:29.604563 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.604487    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 04:54:29.605787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.605761    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-670795-serving-cert\""
Oct 31 04:54:29.708251 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.708205    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5r47\" (UniqueName: \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.708438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.708365    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.708438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.708405    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.708545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.708517    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.780626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.780583    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:29.809818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.809773    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5r47\" (UniqueName: \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.810022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.809893    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.810022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.809927    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.810022 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.809978    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.810724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.810703    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.812554 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.812524    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.812672 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.812652    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.833186 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.833110    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5r47\" (UniqueName: \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\") pod \"maas-api-e2e-inf-a-670795-5747fd846d-ss9zs\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.900248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.898982    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:29.992147 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:29.992076    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:30.214340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.214263    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 04:54:30.217082 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:30.217051    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8278703c_e2f4_48f7_b7d0_7842f5f65a29.slice/crio-3e56436e315234168f2f6e20897847af22e2a18ff6b8ef7cd9470c30f48d7144 WatchSource:0}: Error finding container 3e56436e315234168f2f6e20897847af22e2a18ff6b8ef7cd9470c30f48d7144: Status 404 returned error can't find the container with id 3e56436e315234168f2f6e20897847af22e2a18ff6b8ef7cd9470c30f48d7144
Oct 31 04:54:30.219234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.219213    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:54:30.627181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.619450    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:30.834187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.834134    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:30.850778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.850699    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:30.853865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.853840    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:30.925975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.925935    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" event={"ID":"8278703c-e2f4-48f7-b7d0-7842f5f65a29","Type":"ContainerStarted","Data":"72c34809bd9866369b4cbafad6750f56e8eaad57af5f549a85b299fcaeb9a049"}
Oct 31 04:54:30.926201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.926008    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:30.926201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.926036    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" event={"ID":"8278703c-e2f4-48f7-b7d0-7842f5f65a29","Type":"ContainerStarted","Data":"3e56436e315234168f2f6e20897847af22e2a18ff6b8ef7cd9470c30f48d7144"}
Oct 31 04:54:30.973795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:30.973734    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" podStartSLOduration=1.573611481 podStartE2EDuration="1.973720288s" totalImagesPullingTime="400.108807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:30.970777232 +0000 UTC m=+2967.003776935" watchObservedRunningTime="2026-10-31 04:54:30.973720288 +0000 UTC m=+2967.006719966"
Oct 31 04:54:32.014396 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:32.014341    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:32.020592 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:32.020559    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:32.067006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:32.066967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:32.621932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:32.621892    6931 patch_prober.go:29] interesting pod/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:54:32.622109 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:32.621952    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:54:33.815468 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:33.815432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:34.868632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.868564    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:54:34.955922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.955880    6931 generic.go:356] "Generic (PLEG): container finished" podID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerID="d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413" exitCode=0
Oct 31 04:54:34.955922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.955929    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"
Oct 31 04:54:34.956331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.955950    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" event={"ID":"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d","Type":"ContainerDied","Data":"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"}
Oct 31 04:54:34.956331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.956003    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w" event={"ID":"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d","Type":"ContainerDied","Data":"04fc21f6d90fb9be4c62d4c308d17dff063761d208e265bd3613a409641cb5df"}
Oct 31 04:54:34.956331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.956034    6931 scope.go:122] "RemoveContainer" containerID="d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"
Oct 31 04:54:34.968802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.968773    6931 scope.go:122] "RemoveContainer" containerID="d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"
Oct 31 04:54:34.969172 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:34.969130    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413\": container with ID starting with d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413 not found: ID does not exist" containerID="d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"
Oct 31 04:54:34.969262 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:34.969194    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413"} err="failed to get container status \"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413\": rpc error: code = NotFound desc = could not find container \"d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413\": container with ID starting with d2ec3c66aa5aecbd7d04f34bc8f1acaea0711800c9438d5e1a99c4ad5c2fa413 not found: ID does not exist"
Oct 31 04:54:35.023795 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.023753    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.023999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.023845    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.023999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.023956    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024013    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024044    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:54:35.024128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024060    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024118    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024153    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024237    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024259    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:35.024383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024273    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:54:35.024383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024284    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\") pod \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\" (UID: \"7abc0e64-26b1-4bdb-80bd-27ee4445ad4d\") "
Oct 31 04:54:35.024665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024377    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:54:35.024665 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.024595    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:54:35.025040 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.025017    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.025107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.025048    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.025107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.025066    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.025107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.025080    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.025107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.025095    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.026484 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.026452    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "kube-api-access-7jjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:35.026640 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.026613    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:35.026819 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.026795    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:54:35.026869 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.026836    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy" pod "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" (UID: "7abc0e64-26b1-4bdb-80bd-27ee4445ad4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:54:35.126681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.126589    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.126681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.126624    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjvn\" (UniqueName: \"kubernetes.io/projected/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-kube-api-access-7jjvn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.126681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.126637    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.126681 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.126645    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:35.304426 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.304386    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:54:35.308318 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.308287    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c4006d-openshift-default-7c985cbf7f-b9q2w"]
Oct 31 04:54:35.538884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.538843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:35.851395 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:35.851275    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" podUID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" containerName="payload-processing" containerID="cri-o://15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1" gracePeriod=25
Oct 31 04:54:36.125511 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.125479    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" path="/var/lib/kubelet/pods/7abc0e64-26b1-4bdb-80bd-27ee4445ad4d/volumes"
Oct 31 04:54:36.129301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.129281    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:36.240549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.240512    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\") pod \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") "
Oct 31 04:54:36.240549 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.240551    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\") pod \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") "
Oct 31 04:54:36.240773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.240736    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\") pod \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") "
Oct 31 04:54:36.240850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.240836    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\") pod \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\" (UID: \"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd\") "
Oct 31 04:54:36.241006 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.240980    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume" pod "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" (UID: "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:36.241078 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.241016    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume" pod "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" (UID: "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:36.241590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.241553    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:36.241590 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.241590    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:36.243150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.243125    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials" pod "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" (UID: "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:36.243266 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.243196    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m" pod "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" (UID: "2df4d7ed-7d34-4a08-965c-7e84a3ce36cd"). InnerVolumeSpecName "kube-api-access-pn89m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:36.343281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.343238    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:36.343281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.343274    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn89m\" (UniqueName: \"kubernetes.io/projected/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd-kube-api-access-pn89m\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:36.939172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.939134    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 04:54:36.975784 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.975750    6931 generic.go:356] "Generic (PLEG): container finished" podID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" containerID="15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1" exitCode=0
Oct 31 04:54:36.975957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.975794    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"
Oct 31 04:54:36.975957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.975823    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" event={"ID":"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd","Type":"ContainerDied","Data":"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"}
Oct 31 04:54:36.975957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.975873    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z" event={"ID":"2df4d7ed-7d34-4a08-965c-7e84a3ce36cd","Type":"ContainerDied","Data":"40c8189a9b61044791b00c0c35619244800f9a0a0cb91084f907c360c1bb92c8"}
Oct 31 04:54:36.975957 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.975899    6931 scope.go:122] "RemoveContainer" containerID="15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"
Oct 31 04:54:36.987016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.986998    6931 scope.go:122] "RemoveContainer" containerID="15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"
Oct 31 04:54:36.987348 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:36.987328    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1\": container with ID starting with 15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1 not found: ID does not exist" containerID="15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"
Oct 31 04:54:36.987428 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:36.987354    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1"} err="failed to get container status \"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1\": rpc error: code = NotFound desc = could not find container \"15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1\": container with ID starting with 15821b4ad35e481bd884868d8502cd63a87f26d7bf8ce40944890e4898dc97b1 not found: ID does not exist"
Oct 31 04:54:37.014831 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.014792    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" podUID="d2b274d9-5d25-4298-94dc-80758a2ff93c" containerName="payload-processing" containerID="cri-o://0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3" gracePeriod=25
Oct 31 04:54:37.052199 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.052146    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:37.058070 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.058037    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ded435/payload-processing-external-model-e2e-extmod-b-ded435-5bcdjg75z"]
Oct 31 04:54:37.271188 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.271087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:37.321568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.321538    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:37.459557 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.459518    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\") pod \"d2b274d9-5d25-4298-94dc-80758a2ff93c\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") "
Oct 31 04:54:37.459760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.459648    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\") pod \"d2b274d9-5d25-4298-94dc-80758a2ff93c\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") "
Oct 31 04:54:37.459760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.459678    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\") pod \"d2b274d9-5d25-4298-94dc-80758a2ff93c\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") "
Oct 31 04:54:37.459880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.459786    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\") pod \"d2b274d9-5d25-4298-94dc-80758a2ff93c\" (UID: \"d2b274d9-5d25-4298-94dc-80758a2ff93c\") "
Oct 31 04:54:37.460215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.460187    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume" pod "d2b274d9-5d25-4298-94dc-80758a2ff93c" (UID: "d2b274d9-5d25-4298-94dc-80758a2ff93c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:37.460215 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.460199    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume" pod "d2b274d9-5d25-4298-94dc-80758a2ff93c" (UID: "d2b274d9-5d25-4298-94dc-80758a2ff93c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:54:37.460358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.460326    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:37.460358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.460340    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2b274d9-5d25-4298-94dc-80758a2ff93c-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:37.461924 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.461898    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials" pod "d2b274d9-5d25-4298-94dc-80758a2ff93c" (UID: "d2b274d9-5d25-4298-94dc-80758a2ff93c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:37.462024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.461937    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp" pod "d2b274d9-5d25-4298-94dc-80758a2ff93c" (UID: "d2b274d9-5d25-4298-94dc-80758a2ff93c"). InnerVolumeSpecName "kube-api-access-t2bwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:54:37.561802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.561770    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:37.561802 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.561797    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bwp\" (UniqueName: \"kubernetes.io/projected/d2b274d9-5d25-4298-94dc-80758a2ff93c-kube-api-access-t2bwp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:54:37.984505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.984414    6931 generic.go:356] "Generic (PLEG): container finished" podID="d2b274d9-5d25-4298-94dc-80758a2ff93c" containerID="0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3" exitCode=0
Oct 31 04:54:37.984505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.984466    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"
Oct 31 04:54:37.984505 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.984488    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" event={"ID":"d2b274d9-5d25-4298-94dc-80758a2ff93c","Type":"ContainerDied","Data":"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"}
Oct 31 04:54:37.984740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.984540    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw" event={"ID":"d2b274d9-5d25-4298-94dc-80758a2ff93c","Type":"ContainerDied","Data":"6076dda3cd796a0266fb4b9855b2f66ca52969bd563c7dba609dbbba1d26074c"}
Oct 31 04:54:37.984740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.984569    6931 scope.go:122] "RemoveContainer" containerID="0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"
Oct 31 04:54:37.994961 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.994938    6931 scope.go:122] "RemoveContainer" containerID="0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"
Oct 31 04:54:37.995311 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:37.995283    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3\": container with ID starting with 0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3 not found: ID does not exist" containerID="0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"
Oct 31 04:54:37.995400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:37.995323    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3"} err="failed to get container status \"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3\": rpc error: code = NotFound desc = could not find container \"0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3\": container with ID starting with 0d9ffc9c6444b16f2e9c3d4bf40189aa65fe02646c615d5af9bc19e7ba09dcb3 not found: ID does not exist"
Oct 31 04:54:38.036049 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:38.036017    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:38.047963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:38.047927    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-654741/payload-processing-external-model-e2e-extmod-a-654741-7c745j6kw"]
Oct 31 04:54:38.123759 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:38.123725    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" path="/var/lib/kubelet/pods/2df4d7ed-7d34-4a08-965c-7e84a3ce36cd/volumes"
Oct 31 04:54:38.124146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:38.124132    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b274d9-5d25-4298-94dc-80758a2ff93c" path="/var/lib/kubelet/pods/d2b274d9-5d25-4298-94dc-80758a2ff93c/volumes"
Oct 31 04:54:39.041391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:39.041351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:40.799178 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:54:40.796869    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:54:40.813340 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:40.813301    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:42.693144 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:42.693108    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:43.892495 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:43.892425    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:44.065591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.065552    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:54:44.066280 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066262    6931 cpu_manager.go:414] "containerMap: removing container" podUID="d2b274d9-5d25-4298-94dc-80758a2ff93c" containerName="payload-processing"
Oct 31 04:54:44.066339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066281    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b274d9-5d25-4298-94dc-80758a2ff93c" containerName="payload-processing"
Oct 31 04:54:44.066339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066299    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy"
Oct 31 04:54:44.066339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066305    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy"
Oct 31 04:54:44.066339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066316    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" containerName="payload-processing"
Oct 31 04:54:44.066339 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066322    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" containerName="payload-processing"
Oct 31 04:54:44.066618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066441    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b274d9-5d25-4298-94dc-80758a2ff93c" containerName="payload-processing"
Oct 31 04:54:44.066618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066453    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abc0e64-26b1-4bdb-80bd-27ee4445ad4d" containerName="istio-proxy"
Oct 31 04:54:44.066618 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.066459    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df4d7ed-7d34-4a08-965c-7e84a3ce36cd" containerName="payload-processing"
Oct 31 04:54:44.071675 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.071651    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.089565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.089534    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:54:44.098948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.098905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:44.142239 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.142129    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.142433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.142322    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cj2\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.142433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.142382    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.142433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.142423    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.244436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.244338    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cj2\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.244436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.244396    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.244436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.244424    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.244744 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.244533    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.245130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.245105    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.245268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.245236    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.247066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.247040    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.267285 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.267238    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cj2\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\") pod \"payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") " pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.385273 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.385228    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:44.659513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.659449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:44.807581 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:44.807353    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:54:44.810666 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:54:44.810636    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a09c130_4b81_4d45_8321_b6dd4048435d.slice/crio-bd9c42b8553f04e00d90455a40a860afd1a5f05a4473a79007c8476e068df117 WatchSource:0}: Error finding container bd9c42b8553f04e00d90455a40a860afd1a5f05a4473a79007c8476e068df117: Status 404 returned error can't find the container with id bd9c42b8553f04e00d90455a40a860afd1a5f05a4473a79007c8476e068df117
Oct 31 04:54:45.062240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:45.061850    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" event={"ID":"8a09c130-4b81-4d45-8321-b6dd4048435d","Type":"ContainerStarted","Data":"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"}
Oct 31 04:54:45.062240 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:45.061924    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" event={"ID":"8a09c130-4b81-4d45-8321-b6dd4048435d","Type":"ContainerStarted","Data":"bd9c42b8553f04e00d90455a40a860afd1a5f05a4473a79007c8476e068df117"}
Oct 31 04:54:45.071226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:45.062925    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:45.104830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:45.104774    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" podStartSLOduration=1.104760377 podStartE2EDuration="1.104760377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:54:45.102516711 +0000 UTC m=+2981.135516390" watchObservedRunningTime="2026-10-31 04:54:45.104760377 +0000 UTC m=+2981.137760076"
Oct 31 04:54:46.580538 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:46.580501    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:48.414148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:48.414109    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:50.205055 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:50.205017    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:51.937063 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:51.937005    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:53.720277 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:53.720230    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:53.925853 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:53.925810    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:54.497277 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:54.497231    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:54.828585 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:54.828551    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:55.699389 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:55.699346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:57.078416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:57.078390    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:54:57.150016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:57.149983    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:54:57.478816 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:57.478728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:59.317530 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:54:59.317493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:00.697172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.697103    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:00.886485 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.886450    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 04:55:00.891843 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.891820    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:00.905015 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.904984    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:00.925074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.920145    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-171edd-serving-cert\""
Oct 31 04:55:00.925366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.925340    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 04:55:00.964107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.964016    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:00.964107 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.964059    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:00.964362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.964132    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4k8\" (UniqueName: \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:00.964362 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:00.964231    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.065343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.065291    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.065343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.065345    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.065571 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:01.065453    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-serving-cert: secret "maas-api-e2e-inf-a-171edd-serving-cert" not found
Oct 31 04:55:01.065571 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.065465    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4k8\" (UniqueName: \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.065571 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:01.065519    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls podName:0eb0ef43-32b6-4858-9c03-b309b6fa1d82 nodeName:}" failed. No retries permitted until 2026-10-31 04:55:01.565503598 +0000 UTC m=+2997.598503272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls") pod "maas-api-e2e-inf-a-171edd-b5875b446-scqgc" (UID: "0eb0ef43-32b6-4858-9c03-b309b6fa1d82") : secret "maas-api-e2e-inf-a-171edd-serving-cert" not found
Oct 31 04:55:01.065692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.065578    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.066066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.066043    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.068031 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.068011    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.122309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.122240    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:01.140595 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.140559    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4k8\" (UniqueName: \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.347283 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.347248    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:01.572138 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.572099    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.574798 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.574773    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") pod \"maas-api-e2e-inf-a-171edd-b5875b446-scqgc\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.653130 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.644149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:01.807150 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.807119    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:01.854047 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:01.854016    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:02.155194 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.154499    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" podUID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" containerName="payload-processing" containerID="cri-o://09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091" gracePeriod=25
Oct 31 04:55:02.170727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.169258    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 04:55:02.231670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.227659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:02.461845 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.461816    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:55:02.607473 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.607435    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\") pod \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") "
Oct 31 04:55:02.607624 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.607554    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\") pod \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") "
Oct 31 04:55:02.607700 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.607653    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\") pod \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") "
Oct 31 04:55:02.607756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.607706    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\") pod \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\" (UID: \"1de01a64-bc2a-4272-b68c-b2191c0ec8c2\") "
Oct 31 04:55:02.609793 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.609534    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume" pod "1de01a64-bc2a-4272-b68c-b2191c0ec8c2" (UID: "1de01a64-bc2a-4272-b68c-b2191c0ec8c2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:02.610007 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.609982    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume" pod "1de01a64-bc2a-4272-b68c-b2191c0ec8c2" (UID: "1de01a64-bc2a-4272-b68c-b2191c0ec8c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:02.613813 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.613775    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials" pod "1de01a64-bc2a-4272-b68c-b2191c0ec8c2" (UID: "1de01a64-bc2a-4272-b68c-b2191c0ec8c2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:02.614905 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.614862    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk" pod "1de01a64-bc2a-4272-b68c-b2191c0ec8c2" (UID: "1de01a64-bc2a-4272-b68c-b2191c0ec8c2"). InnerVolumeSpecName "kube-api-access-4x9qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:02.709792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.709748    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:02.709792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.709793    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:02.710057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.709808    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:02.710057 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:02.709822    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x9qk\" (UniqueName: \"kubernetes.io/projected/1de01a64-bc2a-4272-b68c-b2191c0ec8c2-kube-api-access-4x9qk\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:03.056178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.053332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:03.212400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.212359    6931 generic.go:356] "Generic (PLEG): container finished" podID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" containerID="09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091" exitCode=0
Oct 31 04:55:03.212591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.212433    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"
Oct 31 04:55:03.212591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.212443    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" event={"ID":"1de01a64-bc2a-4272-b68c-b2191c0ec8c2","Type":"ContainerDied","Data":"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"}
Oct 31 04:55:03.212591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.212509    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs" event={"ID":"1de01a64-bc2a-4272-b68c-b2191c0ec8c2","Type":"ContainerDied","Data":"15eeef149dbcfeb78e3cf365aca7b17911d4d585dfbfd6931b24984edfe2e459"}
Oct 31 04:55:03.212591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.212544    6931 scope.go:122] "RemoveContainer" containerID="09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"
Oct 31 04:55:03.215024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.214668    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" event={"ID":"0eb0ef43-32b6-4858-9c03-b309b6fa1d82","Type":"ContainerStarted","Data":"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"}
Oct 31 04:55:03.215024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.214720    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" event={"ID":"0eb0ef43-32b6-4858-9c03-b309b6fa1d82","Type":"ContainerStarted","Data":"35491fd359259f15027b6e11fd90def09146e07818ecfdba0081985b322955e7"}
Oct 31 04:55:03.215024 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.214847    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:03.225216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.225193    6931 scope.go:122] "RemoveContainer" containerID="09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"
Oct 31 04:55:03.225654 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:03.225632    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091\": container with ID starting with 09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091 not found: ID does not exist" containerID="09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"
Oct 31 04:55:03.225725 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.225662    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091"} err="failed to get container status \"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091\": rpc error: code = NotFound desc = could not find container \"09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091\": container with ID starting with 09f4916d96f29f52754596b45b5afb383c72fb13bd9aadc4e82f22c5f8835091 not found: ID does not exist"
Oct 31 04:55:03.268985 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.268947    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:55:03.280515 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.280471    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-9f5f2cmvs"]
Oct 31 04:55:03.354635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:03.354523    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" podStartSLOduration=2.9907021670000002 podStartE2EDuration="3.354508192s" totalImagesPullingTime="363.806025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:03.351821867 +0000 UTC m=+2999.384821568" watchObservedRunningTime="2026-10-31 04:55:03.354508192 +0000 UTC m=+2999.387507870"
Oct 31 04:55:04.125429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:04.125389    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" path="/var/lib/kubelet/pods/1de01a64-bc2a-4272-b68c-b2191c0ec8c2/volumes"
Oct 31 04:55:05.295003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:05.294963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:06.167626 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:06.167582    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:06.380543 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:06.380509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:07.325692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:07.325650    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:55:07.336692 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:07.336653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:07.547196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:07.547143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:07.788509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:07.788473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:08.164743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:08.164654    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:08.807472 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:08.807431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:09.229981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:09.229893    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 04:55:09.526731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:09.526645    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:10.212035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:10.211997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:10.475058 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:10.466433    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:10.761994 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:10.761911    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:55:11.314228 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:11.314152    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:12.326687 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.326641    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" podUID="8a09c130-4b81-4d45-8321-b6dd4048435d" containerName="payload-processing" containerID="cri-o://b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e" gracePeriod=25
Oct 31 04:55:12.596204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.596116    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:55:12.691295 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691266    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\") pod \"8a09c130-4b81-4d45-8321-b6dd4048435d\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") "
Oct 31 04:55:12.691498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691430    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\") pod \"8a09c130-4b81-4d45-8321-b6dd4048435d\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") "
Oct 31 04:55:12.691498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691475    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\") pod \"8a09c130-4b81-4d45-8321-b6dd4048435d\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") "
Oct 31 04:55:12.691623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691541    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\") pod \"8a09c130-4b81-4d45-8321-b6dd4048435d\" (UID: \"8a09c130-4b81-4d45-8321-b6dd4048435d\") "
Oct 31 04:55:12.691623 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691607    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume" pod "8a09c130-4b81-4d45-8321-b6dd4048435d" (UID: "8a09c130-4b81-4d45-8321-b6dd4048435d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:12.691881 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.691861    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume" pod "8a09c130-4b81-4d45-8321-b6dd4048435d" (UID: "8a09c130-4b81-4d45-8321-b6dd4048435d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:12.692028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.692009    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:12.692087 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.692030    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a09c130-4b81-4d45-8321-b6dd4048435d-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:12.693695 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.693673    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2" pod "8a09c130-4b81-4d45-8321-b6dd4048435d" (UID: "8a09c130-4b81-4d45-8321-b6dd4048435d"). InnerVolumeSpecName "kube-api-access-d4cj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:12.693788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.693696    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials" pod "8a09c130-4b81-4d45-8321-b6dd4048435d" (UID: "8a09c130-4b81-4d45-8321-b6dd4048435d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:12.793301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.793251    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4cj2\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-kube-api-access-d4cj2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:12.793301 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:12.793291    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a09c130-4b81-4d45-8321-b6dd4048435d-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:13.119416 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.119347    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:13.303382 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.303343    6931 generic.go:356] "Generic (PLEG): container finished" podID="8a09c130-4b81-4d45-8321-b6dd4048435d" containerID="b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e" exitCode=0
Oct 31 04:55:13.303587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.303392    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"
Oct 31 04:55:13.303587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.303404    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" event={"ID":"8a09c130-4b81-4d45-8321-b6dd4048435d","Type":"ContainerDied","Data":"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"}
Oct 31 04:55:13.303587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.303448    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl" event={"ID":"8a09c130-4b81-4d45-8321-b6dd4048435d","Type":"ContainerDied","Data":"bd9c42b8553f04e00d90455a40a860afd1a5f05a4473a79007c8476e068df117"}
Oct 31 04:55:13.303587 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.303468    6931 scope.go:122] "RemoveContainer" containerID="b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"
Oct 31 04:55:13.314146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.314113    6931 scope.go:122] "RemoveContainer" containerID="b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"
Oct 31 04:55:13.314478 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:13.314457    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e\": container with ID starting with b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e not found: ID does not exist" containerID="b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"
Oct 31 04:55:13.314541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.314486    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e"} err="failed to get container status \"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e\": rpc error: code = NotFound desc = could not find container \"b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e\": container with ID starting with b22da27a8c5bfe07830fc789314d70da36e2e14e7b812ff2bc24c4a19c51332e not found: ID does not exist"
Oct 31 04:55:13.354246 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.354211    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:55:13.357982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:13.357953    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-401761/payload-processing-external-model-e2e-extmod-a-401761-6d6cff5jl"]
Oct 31 04:55:14.122949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:14.122915    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a09c130-4b81-4d45-8321-b6dd4048435d" path="/var/lib/kubelet/pods/8a09c130-4b81-4d45-8321-b6dd4048435d/volumes"
Oct 31 04:55:14.240899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:14.240863    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:14.459627 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:14.459543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:14.707750 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:14.707713    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:14.929460 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:14.929424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:15.210337 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:15.210249    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:15.419731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:15.419691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:16.712586 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:16.712533    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:17.253133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:17.253096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:17.461934 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:17.461898    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:18.556513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:18.556477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:18.765558 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:18.765525    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:20.198132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:20.198093    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:20.436901 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:20.436859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:20.934236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:20.934196    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:22.748990 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:22.748950    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:24.695178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:24.695122    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:26.634979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:26.634944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:26.859885 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:26.859846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:27.264517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:27.264473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:27.697218 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:27.697149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:27.908541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:27.908510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:28.551932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:28.551890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:30.410647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:30.410612    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:31.710993 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:31.710957    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:31.940048 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:31.940008    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:32.588327 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:32.588270    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:33.340166 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.340126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:33.417651 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.417614    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 04:55:33.418673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418648    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" containerName="payload-processing"
Oct 31 04:55:33.418765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418675    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" containerName="payload-processing"
Oct 31 04:55:33.418765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418721    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8a09c130-4b81-4d45-8321-b6dd4048435d" containerName="payload-processing"
Oct 31 04:55:33.418765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418731    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a09c130-4b81-4d45-8321-b6dd4048435d" containerName="payload-processing"
Oct 31 04:55:33.418925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418871    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a09c130-4b81-4d45-8321-b6dd4048435d" containerName="payload-processing"
Oct 31 04:55:33.418925 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.418898    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de01a64-bc2a-4272-b68c-b2191c0ec8c2" containerName="payload-processing"
Oct 31 04:55:33.423069 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.423044    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.454224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.454193    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-829ac7\""
Oct 31 04:55:33.454463 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.454256    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-829ac7-dockercfg-49mhb\""
Oct 31 04:55:33.457741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.457714    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 04:55:33.530908 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.530861    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.531074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.531000    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtdt\" (UniqueName: \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.574903 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.574853    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:33.637068 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.633955    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtdt\" (UniqueName: \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.637643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.637608    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.638877 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.638835    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.664238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.664203    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtdt\" (UniqueName: \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\") pod \"payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.741445 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.741411    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:33.833080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.832723    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:55:33.833080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:33.832935    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" podUID="ee0da6dd-8056-4b11-9236-89736c2e809f" containerName="sim" containerID="cri-o://37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818" gracePeriod=30
Oct 31 04:55:34.052772 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.052742    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 04:55:34.056767 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:55:34.056730    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3d97e2_092e_4a1b_8509_94d92225e1b6.slice/crio-11899a19e11925f999ed59ffe92767a25954c577dcdd175ed0feaf7a4d194b36 WatchSource:0}: Error finding container 11899a19e11925f999ed59ffe92767a25954c577dcdd175ed0feaf7a4d194b36: Status 404 returned error can't find the container with id 11899a19e11925f999ed59ffe92767a25954c577dcdd175ed0feaf7a4d194b36
Oct 31 04:55:34.139202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.131326    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:55:34.235564 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.235394    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:34.259702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.258746    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\" (UniqueName: \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\") pod \"ee0da6dd-8056-4b11-9236-89736c2e809f\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") "
Oct 31 04:55:34.259702 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.258860    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\" (UniqueName: \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\") pod \"ee0da6dd-8056-4b11-9236-89736c2e809f\" (UID: \"ee0da6dd-8056-4b11-9236-89736c2e809f\") "
Oct 31 04:55:34.263226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.263194    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls" pod "ee0da6dd-8056-4b11-9236-89736c2e809f" (UID: "ee0da6dd-8056-4b11-9236-89736c2e809f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:55:34.263393 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.263303    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65" pod "ee0da6dd-8056-4b11-9236-89736c2e809f" (UID: "ee0da6dd-8056-4b11-9236-89736c2e809f"). InnerVolumeSpecName "kube-api-access-fmh65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:34.362455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.362408    6931 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ee0da6dd-8056-4b11-9236-89736c2e809f-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:34.362455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.362449    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh65\" (UniqueName: \"kubernetes.io/projected/ee0da6dd-8056-4b11-9236-89736c2e809f-kube-api-access-fmh65\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.506498    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" event={"ID":"7c3d97e2-092e-4a1b-8509-94d92225e1b6","Type":"ContainerStarted","Data":"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"}
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.506566    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" event={"ID":"7c3d97e2-092e-4a1b-8509-94d92225e1b6","Type":"ContainerStarted","Data":"11899a19e11925f999ed59ffe92767a25954c577dcdd175ed0feaf7a4d194b36"}
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.507579    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.509442    6931 generic.go:356] "Generic (PLEG): container finished" podID="ee0da6dd-8056-4b11-9236-89736c2e809f" containerID="37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818" exitCode=0
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.509512    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" event={"ID":"ee0da6dd-8056-4b11-9236-89736c2e809f","Type":"ContainerDied","Data":"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"}
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.509540    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64" event={"ID":"ee0da6dd-8056-4b11-9236-89736c2e809f","Type":"ContainerDied","Data":"90198e782f83b54af73f3cd2b431e521e11ae95d49afbc8161a262e742e7bd67"}
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.509563    6931 scope.go:122] "RemoveContainer" containerID="37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"
Oct 31 04:55:34.515231 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.509710    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"
Oct 31 04:55:34.529369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.526067    6931 scope.go:122] "RemoveContainer" containerID="37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"
Oct 31 04:55:34.529369 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:34.526522    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818\": container with ID starting with 37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818 not found: ID does not exist" containerID="37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"
Oct 31 04:55:34.529369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.526557    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818"} err="failed to get container status \"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818\": rpc error: code = NotFound desc = could not find container \"37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818\": container with ID starting with 37397e96560533ece26029f3ba79e94d53f632d8566aa710af973a272a10e818 not found: ID does not exist"
Oct 31 04:55:34.557097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.556887    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:34.558018 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.557990    6931 cpu_manager.go:414] "containerMap: removing container" podUID="ee0da6dd-8056-4b11-9236-89736c2e809f" containerName="sim"
Oct 31 04:55:34.558120 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.558019    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0da6dd-8056-4b11-9236-89736c2e809f" containerName="sim"
Oct 31 04:55:34.558236 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.558222    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0da6dd-8056-4b11-9236-89736c2e809f" containerName="sim"
Oct 31 04:55:34.577121 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.577057    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" podStartSLOduration=1.577040923 podStartE2EDuration="1.577040923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:34.573736561 +0000 UTC m=+3030.606736241" watchObservedRunningTime="2026-10-31 04:55:34.577040923 +0000 UTC m=+3030.610040604"
Oct 31 04:55:34.578099 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.578076    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.581269 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.581240    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:34.610453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.610407    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:55:34.613746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.613715    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-f5a14c/e2e-usage-sim-04e0c1-6955658f4c-5ws64"]
Oct 31 04:55:34.687529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.676901    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.687529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.677068    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmgb\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.687529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.677222    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.687529 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.677355    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.779281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.779188    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.779443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.779337    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.779443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.779365    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmgb\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.779443 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.779401    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.780086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.780056    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.780241 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.780056    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.781959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.781934    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.802268 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.802241    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmgb\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\") pod \"payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") " pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:34.903114 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:34.903079    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:35.022011 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.020111    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:35.233723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.233698    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:35.236827 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:55:35.236802    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38eb0ba_2eee_4d7b_a6df_70ca15444fd0.slice/crio-dc0927b31a8c4dfea9168ea81c2c14bab4e0fd17e0d1a056043676d4e9c5b543 WatchSource:0}: Error finding container dc0927b31a8c4dfea9168ea81c2c14bab4e0fd17e0d1a056043676d4e9c5b543: Status 404 returned error can't find the container with id dc0927b31a8c4dfea9168ea81c2c14bab4e0fd17e0d1a056043676d4e9c5b543
Oct 31 04:55:35.519913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.519809    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" event={"ID":"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0","Type":"ContainerStarted","Data":"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"}
Oct 31 04:55:35.519913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.519874    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" event={"ID":"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0","Type":"ContainerStarted","Data":"dc0927b31a8c4dfea9168ea81c2c14bab4e0fd17e0d1a056043676d4e9c5b543"}
Oct 31 04:55:35.520459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.519950    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:35.567175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.566685    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"]
Oct 31 04:55:35.574217 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.571181    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" podStartSLOduration=1.5711506750000002 podStartE2EDuration="1.571150675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:35.563380969 +0000 UTC m=+3031.596380649" watchObservedRunningTime="2026-10-31 04:55:35.571150675 +0000 UTC m=+3031.604150356"
Oct 31 04:55:35.574661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.574637    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.593440 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593404    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"]
Oct 31 04:55:35.593610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593487    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"routing-overlay\""
Oct 31 04:55:35.593610 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593570    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"openshift-service-ca.crt\""
Oct 31 04:55:35.593774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593631    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"kube-root-ca.crt\""
Oct 31 04:55:35.593774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593636    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-e2e--ae39f93f-dockercfg-5wzdp\""
Oct 31 04:55:35.593774 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.593720    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a04e4e\""
Oct 31 04:55:35.692569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.692537    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a04e4e\"/\"agc-cred-e77d307cbbfa\""
Oct 31 04:55:35.706656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.698382    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.706656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.698541    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lwr\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.706656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.698749    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.706656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.698798    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.718647 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.707751    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-47c409/agc-cred-d2374fac1bc0: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:35.718647 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.707782    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:35.718647 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.707837    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials podName:d38eb0ba-2eee-4d7b-a6df-70ca15444fd0 nodeName:}" failed. No retries permitted until 2026-10-31 04:55:36.207818348 +0000 UTC m=+3032.240818007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0") : secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:35.733680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.733628    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:35.800190 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.800126    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lwr\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.800433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.800293    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.800433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.800329    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.800433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.800385    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.800630 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.800452    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a04e4e/agc-cred-e77d307cbbfa: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:35.800630 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.800493    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:35.800630 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:35.800573    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials podName:3de3b4e1-6de3-480e-9ba4-57c14422464c nodeName:}" failed. No retries permitted until 2026-10-31 04:55:36.30053806 +0000 UTC m=+3032.333537733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c") : secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:35.801242 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.801220    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.801328 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.801252    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.826196 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.826142    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lwr\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:35.948691 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:35.948653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:36.124588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:36.124511    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0da6dd-8056-4b11-9236-89736c2e809f" path="/var/lib/kubelet/pods/ee0da6dd-8056-4b11-9236-89736c2e809f/volumes"
Oct 31 04:55:36.302072 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:36.302033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:36.306533 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:36.306505    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:36.306533 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306532    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-47c409/agc-cred-d2374fac1bc0: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:36.306762 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306549    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:36.306762 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306598    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials podName:d38eb0ba-2eee-4d7b-a6df-70ca15444fd0 nodeName:}" failed. No retries permitted until 2026-10-31 04:55:37.306583454 +0000 UTC m=+3033.339583111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0") : secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:36.306762 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306641    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a04e4e/agc-cred-e77d307cbbfa: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:36.306762 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306660    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:36.306762 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:36.306730    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials podName:3de3b4e1-6de3-480e-9ba4-57c14422464c nodeName:}" failed. No retries permitted until 2026-10-31 04:55:37.306712806 +0000 UTC m=+3033.339712477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c") : secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:36.834949 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:36.834918    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:55:36.846731 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:36.846695    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:37.002907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:37.002870    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:37.223656 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:37.223558    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:37.320400 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:37.320366    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:37.320616 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320516    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a04e4e/agc-cred-e77d307cbbfa: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:37.320616 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320534    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:37.320616 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320583    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials podName:3de3b4e1-6de3-480e-9ba4-57c14422464c nodeName:}" failed. No retries permitted until 2026-10-31 04:55:39.32056759 +0000 UTC m=+3035.353567247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c") : secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:37.320785 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320632    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-47c409/agc-cred-d2374fac1bc0: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:37.320785 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320650    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:37.320785 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.320704    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials podName:d38eb0ba-2eee-4d7b-a6df-70ca15444fd0 nodeName:}" failed. No retries permitted until 2026-10-31 04:55:39.320688206 +0000 UTC m=+3035.353687865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0") : secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:37.757616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:37.757582    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"]
Oct 31 04:55:37.757932 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:37.757913    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd" podUID="3de3b4e1-6de3-480e-9ba4-57c14422464c"
Oct 31 04:55:37.939077 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:37.939039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:38.570709 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.570674    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:38.579122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.579098    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:38.751272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.748909    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\") pod \"3de3b4e1-6de3-480e-9ba4-57c14422464c\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") "
Oct 31 04:55:38.751272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.748965    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\") pod \"3de3b4e1-6de3-480e-9ba4-57c14422464c\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") "
Oct 31 04:55:38.751272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.749025    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\") pod \"3de3b4e1-6de3-480e-9ba4-57c14422464c\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") "
Oct 31 04:55:38.751272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.750226    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume" pod "3de3b4e1-6de3-480e-9ba4-57c14422464c" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:38.760616 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.760363    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr" pod "3de3b4e1-6de3-480e-9ba4-57c14422464c" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c"). InnerVolumeSpecName "kube-api-access-65lwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:38.760786 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.760678    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume" pod "3de3b4e1-6de3-480e-9ba4-57c14422464c" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:38.798349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.798302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:38.850788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.850699    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:38.850788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.850734    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lwr\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-kube-api-access-65lwr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:38.850788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:38.850748    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3de3b4e1-6de3-480e-9ba4-57c14422464c-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:39.357767 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:39.357725    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd\" (UID: \"3de3b4e1-6de3-480e-9ba4-57c14422464c\") " pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.357891    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-a04e4e/agc-cred-e77d307cbbfa: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.357920    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd: secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.357952    6931 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-47c409/agc-cred-d2374fac1bc0: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.357965    6931 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd: secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.357976    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials podName:3de3b4e1-6de3-480e-9ba4-57c14422464c nodeName:}" failed. No retries permitted until 2026-10-31 04:55:43.357960866 +0000 UTC m=+3039.390960523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd" (UID: "3de3b4e1-6de3-480e-9ba4-57c14422464c") : secret "agc-cred-e77d307cbbfa" not found
Oct 31 04:55:39.358252 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:39.358002    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials podName:d38eb0ba-2eee-4d7b-a6df-70ca15444fd0 nodeName:}" failed. No retries permitted until 2026-10-31 04:55:43.357991482 +0000 UTC m=+3039.390991139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0") : secret "agc-cred-d2374fac1bc0" not found
Oct 31 04:55:39.576824 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:39.576790    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"
Oct 31 04:55:39.651270 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:39.651182    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"]
Oct 31 04:55:39.660666 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:39.660631    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a04e4e/payload-processing-external-model-e2e-extmod-b-a04e4e-796drnxnd"]
Oct 31 04:55:39.763084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:39.763045    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3de3b4e1-6de3-480e-9ba4-57c14422464c-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:40.124086 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:40.124053    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de3b4e1-6de3-480e-9ba4-57c14422464c" path="/var/lib/kubelet/pods/3de3b4e1-6de3-480e-9ba4-57c14422464c/volumes"
Oct 31 04:55:40.616370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:40.616332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:40.759983 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:40.759936    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:55:41.835781 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:41.835736    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" podUID="8391ba07-9a69-4e6f-9b8d-44a63069334f" containerName="payload-processing" containerID="cri-o://46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad" gracePeriod=25
Oct 31 04:55:42.103632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.103568    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:55:42.192591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.192555    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\") pod \"8391ba07-9a69-4e6f-9b8d-44a63069334f\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") "
Oct 31 04:55:42.192792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.192723    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\") pod \"8391ba07-9a69-4e6f-9b8d-44a63069334f\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") "
Oct 31 04:55:42.192792 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.192763    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\") pod \"8391ba07-9a69-4e6f-9b8d-44a63069334f\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") "
Oct 31 04:55:42.192922 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.192819    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\") pod \"8391ba07-9a69-4e6f-9b8d-44a63069334f\" (UID: \"8391ba07-9a69-4e6f-9b8d-44a63069334f\") "
Oct 31 04:55:42.193363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.193335    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume" pod "8391ba07-9a69-4e6f-9b8d-44a63069334f" (UID: "8391ba07-9a69-4e6f-9b8d-44a63069334f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:42.193539 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.193368    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume" pod "8391ba07-9a69-4e6f-9b8d-44a63069334f" (UID: "8391ba07-9a69-4e6f-9b8d-44a63069334f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:42.194948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.194923    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv" pod "8391ba07-9a69-4e6f-9b8d-44a63069334f" (UID: "8391ba07-9a69-4e6f-9b8d-44a63069334f"). InnerVolumeSpecName "kube-api-access-q2pvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:42.195243 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.195221    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials" pod "8391ba07-9a69-4e6f-9b8d-44a63069334f" (UID: "8391ba07-9a69-4e6f-9b8d-44a63069334f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:42.295221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.295186    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pvv\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-kube-api-access-q2pvv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:42.295221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.295218    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:42.295221 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.295228    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8391ba07-9a69-4e6f-9b8d-44a63069334f-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:42.295469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.295238    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8391ba07-9a69-4e6f-9b8d-44a63069334f-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:42.538234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.538190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:42.570974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.570300    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" podUID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" containerName="payload-processing" containerID="cri-o://4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925" gracePeriod=25
Oct 31 04:55:42.607117 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.607082    6931 generic.go:356] "Generic (PLEG): container finished" podID="8391ba07-9a69-4e6f-9b8d-44a63069334f" containerID="46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad" exitCode=0
Oct 31 04:55:42.607348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.607135    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" event={"ID":"8391ba07-9a69-4e6f-9b8d-44a63069334f","Type":"ContainerDied","Data":"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"}
Oct 31 04:55:42.607348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.607147    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"
Oct 31 04:55:42.607348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.607197    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k" event={"ID":"8391ba07-9a69-4e6f-9b8d-44a63069334f","Type":"ContainerDied","Data":"18dff8e69c20b930638a078b2a653f2c9ec82e363292047c9397e8f76e5c7d61"}
Oct 31 04:55:42.607348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.607228    6931 scope.go:122] "RemoveContainer" containerID="46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"
Oct 31 04:55:42.683448 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.683409    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:55:42.690475 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.690454    6931 scope.go:122] "RemoveContainer" containerID="46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"
Oct 31 04:55:42.690866 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:42.690846    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad\": container with ID starting with 46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad not found: ID does not exist" containerID="46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"
Oct 31 04:55:42.690942 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.690875    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad"} err="failed to get container status \"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad\": rpc error: code = NotFound desc = could not find container \"46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad\": container with ID starting with 46aa21d5d757f55953f8bcf2a18f2f8eb5fcc874e7d080a097f6a474433e81ad not found: ID does not exist"
Oct 31 04:55:42.695483 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.695459    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-5f467949k"]
Oct 31 04:55:42.790461 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.790358    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:42.849423 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.849396    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:42.905779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.905728    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\") pod \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") "
Oct 31 04:55:42.905948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.905792    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\") pod \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") "
Oct 31 04:55:42.905948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.905887    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\") pod \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") "
Oct 31 04:55:42.906059 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.906042    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\") pod \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\" (UID: \"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0\") "
Oct 31 04:55:42.906309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.906284    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume" pod "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:42.906614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.906591    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:42.907025 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.906999    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume" pod "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:55:42.913197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.908469    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials" pod "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:42.913197 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.908928    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb" pod "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" (UID: "d38eb0ba-2eee-4d7b-a6df-70ca15444fd0"). InnerVolumeSpecName "kube-api-access-vvmgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:55:42.995020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:42.994981    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:43.007629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.007599    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmgb\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-kube-api-access-vvmgb\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:43.007629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.007627    6931 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-provider-credentials\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:43.007778 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.007637    6931 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0-routing-overlay-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 04:55:43.242237 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.242113    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:43.619884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.619844    6931 generic.go:356] "Generic (PLEG): container finished" podID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" containerID="4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925" exitCode=0
Oct 31 04:55:43.619884 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.619894    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"
Oct 31 04:55:43.620214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.619935    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" event={"ID":"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0","Type":"ContainerDied","Data":"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"}
Oct 31 04:55:43.620214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.620007    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd" event={"ID":"d38eb0ba-2eee-4d7b-a6df-70ca15444fd0","Type":"ContainerDied","Data":"dc0927b31a8c4dfea9168ea81c2c14bab4e0fd17e0d1a056043676d4e9c5b543"}
Oct 31 04:55:43.620214 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.620039    6931 scope.go:122] "RemoveContainer" containerID="4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"
Oct 31 04:55:43.632989 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.632965    6931 scope.go:122] "RemoveContainer" containerID="4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"
Oct 31 04:55:43.633328 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:55:43.633304    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925\": container with ID starting with 4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925 not found: ID does not exist" containerID="4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"
Oct 31 04:55:43.633399 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.633339    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925"} err="failed to get container status \"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925\": rpc error: code = NotFound desc = could not find container \"4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925\": container with ID starting with 4a5d873db2d2cbecf93db0b8d0bb2aa42e9fbe9c4b80cf3ae01fbc3fbb4b3925 not found: ID does not exist"
Oct 31 04:55:43.714013 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.713970    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:43.728384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.728008    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-47c409/payload-processing-external-model-e2e-extmod-a-47c409-6b6c6jtpd"]
Oct 31 04:55:43.764177 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.751816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:43.974510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:43.974416    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:44.124996 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:44.124961    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8391ba07-9a69-4e6f-9b8d-44a63069334f" path="/var/lib/kubelet/pods/8391ba07-9a69-4e6f-9b8d-44a63069334f/volumes"
Oct 31 04:55:44.125435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:44.125419    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" path="/var/lib/kubelet/pods/d38eb0ba-2eee-4d7b-a6df-70ca15444fd0/volumes"
Oct 31 04:55:44.482683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:44.482631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:46.331287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:46.331233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:46.561370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:46.561340    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 04:55:47.398338 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:47.398302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:47.716431 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:47.716336    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:48.428409 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:48.428373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:49.136249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.136212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:49.347226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.344689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:49.456148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.456060    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 04:55:49.457103 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457080    6931 cpu_manager.go:414] "containerMap: removing container" podUID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" containerName="payload-processing"
Oct 31 04:55:49.457179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457107    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" containerName="payload-processing"
Oct 31 04:55:49.457222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457191    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8391ba07-9a69-4e6f-9b8d-44a63069334f" containerName="payload-processing"
Oct 31 04:55:49.457222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457201    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8391ba07-9a69-4e6f-9b8d-44a63069334f" containerName="payload-processing"
Oct 31 04:55:49.457366 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457351    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8391ba07-9a69-4e6f-9b8d-44a63069334f" containerName="payload-processing"
Oct 31 04:55:49.457414 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.457378    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38eb0ba-2eee-4d7b-a6df-70ca15444fd0" containerName="payload-processing"
Oct 31 04:55:49.462469 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.462446    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.483021 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.482993    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 04:55:49.491568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.491544    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1daf30-dockercfg-j8pn5\""
Oct 31 04:55:49.491568 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.491544    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1daf30\""
Oct 31 04:55:49.557491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.557394    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh88c\" (UniqueName: \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.557491 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.557463    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.649619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.644872    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 04:55:49.651364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.650184    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.659370 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.659340    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh88c\" (UniqueName: \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.659671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.659647    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.660647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.660625    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.676805 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.676769    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 04:55:49.697281 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.697230    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh88c\" (UniqueName: \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\") pod \"payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.761168 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.761077    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.761312 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.761244    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn8c\" (UniqueName: \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.777763 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.777734    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:49.863288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.862173    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn8c\" (UniqueName: \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.863288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.862385    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.863288 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.863246    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.894787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.894749    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn8c\" (UniqueName: \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\") pod \"payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:49.977969 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:49.977932    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:50.098075 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.097120    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 04:55:50.289391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.289315    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 04:55:50.293456 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:55:50.293430    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7597c178_3922_4646_9ee2_75b5474fec53.slice/crio-70ebf5358133d13a2af123b5728c9faa0eeb6d26704bcc08358735cf60924344 WatchSource:0}: Error finding container 70ebf5358133d13a2af123b5728c9faa0eeb6d26704bcc08358735cf60924344: Status 404 returned error can't find the container with id 70ebf5358133d13a2af123b5728c9faa0eeb6d26704bcc08358735cf60924344
Oct 31 04:55:50.672724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.672622    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" event={"ID":"200cdbab-5ef1-406c-9397-b89894111a40","Type":"ContainerStarted","Data":"87cbcfd7c2734e59cfc33fdaa0eae8561c3789be63c19684bc41f8de9efe9d15"}
Oct 31 04:55:50.672724 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.672684    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" event={"ID":"200cdbab-5ef1-406c-9397-b89894111a40","Type":"ContainerStarted","Data":"40b844b6d02af8d989e355ad02ec80ea9d37ad37465246e9f80ea6ace3fd3af3"}
Oct 31 04:55:50.673265 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.672848    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:55:50.674569 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.674543    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" event={"ID":"7597c178-3922-4646-9ee2-75b5474fec53","Type":"ContainerStarted","Data":"463e3fcdd6c14e511402f9642fb0c933aef9c7f5116bfe3c5521298391460013"}
Oct 31 04:55:50.674676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.674593    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:55:50.674676 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.674610    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" event={"ID":"7597c178-3922-4646-9ee2-75b5474fec53","Type":"ContainerStarted","Data":"70ebf5358133d13a2af123b5728c9faa0eeb6d26704bcc08358735cf60924344"}
Oct 31 04:55:50.712010 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.711957    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" podStartSLOduration=1.711943447 podStartE2EDuration="1.711943447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:50.711075196 +0000 UTC m=+3046.744074876" watchObservedRunningTime="2026-10-31 04:55:50.711943447 +0000 UTC m=+3046.744943162"
Oct 31 04:55:50.752800 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.752743    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" podStartSLOduration=1.752729135 podStartE2EDuration="1.752729135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:55:50.749448743 +0000 UTC m=+3046.782448475" watchObservedRunningTime="2026-10-31 04:55:50.752729135 +0000 UTC m=+3046.785728833"
Oct 31 04:55:50.997257 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:50.993974    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:53.222777 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:53.222740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:55.009953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:55.009913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:56.732474 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:56.732439    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:58.782643 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:55:58.782604    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:00.476369 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.476334    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 04:56:00.480978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.480960    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.497385 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.497355    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-108ad8-serving-cert\""
Oct 31 04:56:00.513756 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.513709    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 04:56:00.519809 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.519771    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.519982 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.519937    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqbv\" (UniqueName: \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.520066 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.520042    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.520151 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.520135    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.621763 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.621724    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.621963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.621839    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqbv\" (UniqueName: \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.621963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.621897    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.622084 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.621964    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.622137 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:00.622120    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-serving-cert: secret "maas-api-e2e-inf-a-108ad8-serving-cert" not found
Oct 31 04:56:00.622240 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:00.622214    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls podName:529a5853-adf2-4250-96d1-9a9fc9d251e2 nodeName:}" failed. No retries permitted until 2026-10-31 04:56:01.12219404 +0000 UTC m=+3057.155193697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls") pod "maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" (UID: "529a5853-adf2-4250-96d1-9a9fc9d251e2") : secret "maas-api-e2e-inf-a-108ad8-serving-cert" not found
Oct 31 04:56:00.622594 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.622573    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.624386 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.624366    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.645128 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.645101    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqbv\" (UniqueName: \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:00.699003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.698963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:00.910514 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:00.910473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:01.118791 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.118757    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:01.129149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.129122    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:01.131551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.131530    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:01.396412 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.396374    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:01.684697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.684607    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 04:56:01.684697 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.684683    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 04:56:01.743811 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.743780    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 04:56:01.748914 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:56:01.748885    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529a5853_adf2_4250_96d1_9a9fc9d251e2.slice/crio-235176bbc192aa7b4d72e1ff1d618b48f6b93c24649f4bba19bf99c262be1bb5 WatchSource:0}: Error finding container 235176bbc192aa7b4d72e1ff1d618b48f6b93c24649f4bba19bf99c262be1bb5: Status 404 returned error can't find the container with id 235176bbc192aa7b4d72e1ff1d618b48f6b93c24649f4bba19bf99c262be1bb5
Oct 31 04:56:01.781253 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.781213    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" event={"ID":"529a5853-adf2-4250-96d1-9a9fc9d251e2","Type":"ContainerStarted","Data":"235176bbc192aa7b4d72e1ff1d618b48f6b93c24649f4bba19bf99c262be1bb5"}
Oct 31 04:56:01.849620 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:01.849579    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:02.060091 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.060054    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:02.438135 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.438035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:02.793455 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.793413    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" event={"ID":"529a5853-adf2-4250-96d1-9a9fc9d251e2","Type":"ContainerStarted","Data":"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"}
Oct 31 04:56:02.793964 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.793721    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:02.841195 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.841060    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" podStartSLOduration=2.487869381 podStartE2EDuration="2.841041408s" totalImagesPullingTime="353.172027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:02.838730851 +0000 UTC m=+3058.871730550" watchObservedRunningTime="2026-10-31 04:56:02.841041408 +0000 UTC m=+3058.874041089"
Oct 31 04:56:02.843541 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:02.843509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:03.057625 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:03.057535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:04.966711 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:04.966666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:06.910331 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:06.910289    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:07.163065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:07.162967    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:07.390105 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:07.390069    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:08.807381 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.807345    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 04:56:08.837542 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.837508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:08.934213 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.934175    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 04:56:08.939822 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.939796    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:08.956187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.956135    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fb346b\""
Oct 31 04:56:08.970826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.963292    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fb346b-dockercfg-kjqf5\""
Oct 31 04:56:08.970826 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:08.967782    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 04:56:09.038435 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.038392    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.049179 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.039100    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtb2\" (UniqueName: \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.074647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.070399    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 04:56:09.074647 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.073325    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:09.100913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.099543    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 04:56:09.100913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.099701    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.161085 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.161047    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.161315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.161116    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtb2\" (UniqueName: \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.161315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.161183    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.161315 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.161264    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5fb\" (UniqueName: \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.161914 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.161891    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.184579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.184550    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtb2\" (UniqueName: \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\") pod \"payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.262223 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.262189    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.262417 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.262297    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5fb\" (UniqueName: \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.263042 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.263016    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.272974 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.272949    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.285497 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.285470    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5fb\" (UniqueName: \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\") pod \"payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.427267 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.427234    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.583026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.583000    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 04:56:09.586232 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:56:09.586208    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e01560_0500_47bd_8070_4c7e2a50b76c.slice/crio-6e5b2c984e28e9432c4e3aefacc6250cb038b58e5236a43ae6e3af7ab901ae46 WatchSource:0}: Error finding container 6e5b2c984e28e9432c4e3aefacc6250cb038b58e5236a43ae6e3af7ab901ae46: Status 404 returned error can't find the container with id 6e5b2c984e28e9432c4e3aefacc6250cb038b58e5236a43ae6e3af7ab901ae46
Oct 31 04:56:09.734220 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.734195    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 04:56:09.737846 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:56:09.737818    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7f1f45_03a8_4c35_9c8d_c3c9c287742a.slice/crio-2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09 WatchSource:0}: Error finding container 2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09: Status 404 returned error can't find the container with id 2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09
Oct 31 04:56:09.842895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.842817    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" event={"ID":"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a","Type":"ContainerStarted","Data":"63bfe7bd046ab1e439d91c1bf18bdb412458a8ea0e2554aaca9d3f84a35e8043"}
Oct 31 04:56:09.842895 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.842868    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" event={"ID":"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a","Type":"ContainerStarted","Data":"2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09"}
Oct 31 04:56:09.843436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.842937    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:09.844537 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.844515    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" event={"ID":"98e01560-0500-47bd-8070-4c7e2a50b76c","Type":"ContainerStarted","Data":"cbb01d5f11964e197711c63d33fca6b2176a635c86fc5f397d64a1c5378b7833"}
Oct 31 04:56:09.844649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.844561    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" event={"ID":"98e01560-0500-47bd-8070-4c7e2a50b76c","Type":"ContainerStarted","Data":"6e5b2c984e28e9432c4e3aefacc6250cb038b58e5236a43ae6e3af7ab901ae46"}
Oct 31 04:56:09.844649 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.844593    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:09.884570 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.884519    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" podStartSLOduration=1.884505039 podStartE2EDuration="1.884505039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:09.882011871 +0000 UTC m=+3065.915011569" watchObservedRunningTime="2026-10-31 04:56:09.884505039 +0000 UTC m=+3065.917504748"
Oct 31 04:56:09.923204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:09.923132    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" podStartSLOduration=1.9231199559999999 podStartE2EDuration="1.923119956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:09.9219142 +0000 UTC m=+3065.954913895" watchObservedRunningTime="2026-10-31 04:56:09.923119956 +0000 UTC m=+3065.956119634"
Oct 31 04:56:10.759633 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:10.759596    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:56:10.876768 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:10.876730    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:12.814913 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:12.814866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:14.724863 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:14.724781    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:15.162840 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:15.162803    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:16.361517 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.361482    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 04:56:16.366454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.366434    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.382038 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.382005    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04d111-serving-cert\""
Oct 31 04:56:16.400026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.399995    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 04:56:16.457810 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.457777    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.457999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.457824    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.457999 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.457864    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztw4s\" (UniqueName: \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.458106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.458057    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.570789 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.570746    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.571071 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.571050    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.571226 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.571208    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztw4s\" (UniqueName: \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.571589 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.571565    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.575929 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.575884    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:16.576859 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.576827    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.589302 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.589200    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.589788 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.589758    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.608500 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.608470    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztw4s\" (UniqueName: \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\") pod \"maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.681561 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.681470    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:16.788132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.788085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:16.988134 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:16.988104    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 04:56:16.990426 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:56:16.990401    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02c0998_b492_4575_ba14_1bf8076fa354.slice/crio-b88109387ae343d4ce561fd37b14bdf14698017d2a67d4e7a4c8e29fc51ecb3d WatchSource:0}: Error finding container b88109387ae343d4ce561fd37b14bdf14698017d2a67d4e7a4c8e29fc51ecb3d: Status 404 returned error can't find the container with id b88109387ae343d4ce561fd37b14bdf14698017d2a67d4e7a4c8e29fc51ecb3d
Oct 31 04:56:17.902407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:17.902370    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" event={"ID":"c02c0998-b492-4575-ba14-1bf8076fa354","Type":"ContainerStarted","Data":"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"}
Oct 31 04:56:17.902890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:17.902430    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" event={"ID":"c02c0998-b492-4575-ba14-1bf8076fa354","Type":"ContainerStarted","Data":"b88109387ae343d4ce561fd37b14bdf14698017d2a67d4e7a4c8e29fc51ecb3d"}
Oct 31 04:56:17.902890 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:17.902539    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:18.408444 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:18.408398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:19.450719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:19.450677    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:20.273943 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:20.273909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:20.856634 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:20.856605    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 04:56:20.857030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:20.856673    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 04:56:20.900373 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:20.900307    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" podStartSLOduration=4.551147099 podStartE2EDuration="4.900288107s" totalImagesPullingTime="349.141008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:17.952491626 +0000 UTC m=+3073.985491306" watchObservedRunningTime="2026-10-31 04:56:20.900288107 +0000 UTC m=+3076.933287787"
Oct 31 04:56:22.112997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:22.112963    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:23.932003 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:23.931968    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 04:56:23.968090 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:23.968041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:25.777719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:25.777679    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:27.626861 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:27.626798    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:29.163493 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.163455    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 04:56:29.169376 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.169353    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.188319 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.188286    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 04:56:29.195868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.195833    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-829ac7-serving-cert\""
Oct 31 04:56:29.306773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.306738    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.306963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.306877    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.306963 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.306924    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.307097 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.307026    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtbg\" (UniqueName: \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.358439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.358396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:29.408498 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.408459    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.408707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.408528    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.408707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.408616    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtbg\" (UniqueName: \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.408838 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.408738    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.409438 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.409404    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.413175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.412008    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.413175 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.412053    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.445238 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.438288    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtbg\" (UniqueName: \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\") pod \"maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.485344 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.485061    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:29.582953 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.582923    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:29.840453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.840429    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 04:56:29.843471 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:56:29.843446    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda4ea64_5dd6_421e_bfc5_e3b290990ca6.slice/crio-290ead6ff9932d3a06daa90651f8daa530cd8018456fc1f7f3a8015b66699530 WatchSource:0}: Error finding container 290ead6ff9932d3a06daa90651f8daa530cd8018456fc1f7f3a8015b66699530: Status 404 returned error can't find the container with id 290ead6ff9932d3a06daa90651f8daa530cd8018456fc1f7f3a8015b66699530
Oct 31 04:56:29.986596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:29.986557    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" event={"ID":"dda4ea64-5dd6-421e-bfc5-e3b290990ca6","Type":"ContainerStarted","Data":"290ead6ff9932d3a06daa90651f8daa530cd8018456fc1f7f3a8015b66699530"}
Oct 31 04:56:30.995389 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:30.995345    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" event={"ID":"dda4ea64-5dd6-421e-bfc5-e3b290990ca6","Type":"ContainerStarted","Data":"4f29953be3add03a4dd0c45a8982e5759579b2f0fda52ae66dacd4c3cd9095cd"}
Oct 31 04:56:30.995846 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:30.995566    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:31.038655 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:31.038600    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" podStartSLOduration=1.713515431 podStartE2EDuration="2.038585674s" totalImagesPullingTime="325.070243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:31.036432966 +0000 UTC m=+3087.069432666" watchObservedRunningTime="2026-10-31 04:56:31.038585674 +0000 UTC m=+3087.071585352"
Oct 31 04:56:31.409997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:31.409954    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:33.274551 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:33.274518    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:35.127883 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:35.127844    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:36.928494 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:36.928449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:37.008143 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:37.008111    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 04:56:38.681119 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:38.681067    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:40.516918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:40.516867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:40.760790 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:40.760745    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:56:42.296133 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:42.296094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:44.091841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:44.091799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:45.757372 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.757340    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 04:56:45.762652 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.762632    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.790471 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.790437    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1daf30-serving-cert\""
Oct 31 04:56:45.793657 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.793629    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 04:56:45.870710 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.870670    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzljw\" (UniqueName: \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.870893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.870735    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.870893 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.870845    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.871026 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.870988    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.972700    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzljw\" (UniqueName: \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.972830    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.972933    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.973062    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:45.979327    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-serving-cert: secret "maas-api-e2e-inf-a-1daf30-serving-cert" not found
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:56:45.979420    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls podName:4e483801-b8a1-44a1-8999-efd4a8faf25d nodeName:}" failed. No retries permitted until 2026-10-31 04:56:46.479397962 +0000 UTC m=+3102.512397635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls") pod "maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" (UID: "4e483801-b8a1-44a1-8999-efd4a8faf25d") : secret "maas-api-e2e-inf-a-1daf30-serving-cert" not found
Oct 31 04:56:45.983181 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.980603    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.983827 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.983405    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:45.986459 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:45.984807    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:46.003871 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.003821    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzljw\" (UniqueName: \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:46.198753 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.198701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:46.421429 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.421380    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:46.479970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.479890    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:46.482397 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.482374    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") pod \"maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:46.678363 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:46.678266    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:47.014512 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:47.009597    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 04:56:47.116364 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:47.116330    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" event={"ID":"4e483801-b8a1-44a1-8999-efd4a8faf25d","Type":"ContainerStarted","Data":"8f452bb9d5a218c8d2ff447706c7d6d99a7f62b3123ac1702e65dbffdebab604"}
Oct 31 04:56:47.854741 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:47.854702    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:48.124979 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:48.124898    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" event={"ID":"4e483801-b8a1-44a1-8999-efd4a8faf25d","Type":"ContainerStarted","Data":"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"}
Oct 31 04:56:48.125379 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:48.125029    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:48.171035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:48.170978    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" podStartSLOduration=2.82356 podStartE2EDuration="3.170965194s" totalImagesPullingTime="347.405194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:56:48.167914307 +0000 UTC m=+3104.200913999" watchObservedRunningTime="2026-10-31 04:56:48.170965194 +0000 UTC m=+3104.203964872"
Oct 31 04:56:49.642936 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:49.642883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:51.418743 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:51.418697    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:53.266439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:53.266383    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:54.137216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:54.137182    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 04:56:55.227353 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:55.227300    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:57.139988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:57.139947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:58.993028 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:56:58.992993    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:00.888605 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:00.888545    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:01.969291 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:01.969230    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:02.906841 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:02.906798    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:03.993749 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:03.993710    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:04.471248 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:04.471212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:04.824694 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:04.824573    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:05.057736 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:05.057700    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:05.283712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:05.283680    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:06.449096 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.449060    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 04:57:06.453754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.453734    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.472680 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.472632    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 04:57:06.480602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.480570    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fb346b-serving-cert\""
Oct 31 04:57:06.623305 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.623271    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.623499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.623387    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.623499 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.623434    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcspn\" (UniqueName: \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.623648 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.623624    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.639433 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.639399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:06.725462 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.725330    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.726358 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.726326    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.726619 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.726596    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.726799 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.726780    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcspn\" (UniqueName: \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.739880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.739789    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.741287 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.741257    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.751098 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.751065    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.753330 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.753300    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcspn\" (UniqueName: \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\") pod \"maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.767407 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.767374    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:06.854980 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:06.854932    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:07.079297 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:07.079232    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 04:57:07.081987 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:57:07.081964    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0a447a_fa13_43a3_8d81_6191d115ff17.slice/crio-af869de835132ee989ded88e24e2481a2a0b058bd5999f8e21ecc4ca4ef4a10c WatchSource:0}: Error finding container af869de835132ee989ded88e24e2481a2a0b058bd5999f8e21ecc4ca4ef4a10c: Status 404 returned error can't find the container with id af869de835132ee989ded88e24e2481a2a0b058bd5999f8e21ecc4ca4ef4a10c
Oct 31 04:57:07.288907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:07.288864    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" event={"ID":"0c0a447a-fa13-43a3-8d81-6191d115ff17","Type":"ContainerStarted","Data":"af869de835132ee989ded88e24e2481a2a0b058bd5999f8e21ecc4ca4ef4a10c"}
Oct 31 04:57:08.297821 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:08.297777    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" event={"ID":"0c0a447a-fa13-43a3-8d81-6191d115ff17","Type":"ContainerStarted","Data":"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"}
Oct 31 04:57:08.298528 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:08.297977    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:08.339918 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:08.339856    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" podStartSLOduration=1.922876791 podStartE2EDuration="2.339840367s" totalImagesPullingTime="416.963576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:57:08.339107903 +0000 UTC m=+3124.372107584" watchObservedRunningTime="2026-10-31 04:57:08.339840367 +0000 UTC m=+3124.372840064"
Oct 31 04:57:08.616670 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:08.616576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:10.230230 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:10.230146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:10.443723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:10.443673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:10.652614 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:10.652581    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:10.766147 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:57:10.766103    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:57:12.040773 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:12.040728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:12.271567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:12.271526    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:12.915673 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:12.915638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:14.321349 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:14.321313    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 04:57:14.815272 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:14.815237    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:16.533016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:16.532976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:18.410509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:18.410474    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:20.456127 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:20.456084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:21.268740 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:21.268704    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:21.712314 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:21.712275    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:22.302427 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:22.302335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:22.527981 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:22.527937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:22.741932 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:22.741774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:24.514867 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:24.514818    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:26.406313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:26.406272    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:26.615727 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:26.615694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:28.118790 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:28.118752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:28.325947 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:28.325915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:28.569222 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:28.569189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:30.492526 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:30.492485    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:32.266020 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:32.265983    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:35.849212 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:35.849174    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:36.081348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:36.081304    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:37.796436 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:37.796393    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:39.615565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:39.615523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:39.848216 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:39.848177    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:40.761503 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:57:40.761459    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:57:41.460959 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:41.460925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:41.672785 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:41.672735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:43.292136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:43.292088    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:45.272712 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:45.269315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:47.297348 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:47.297308    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:49.262122 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:49.262071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:51.355923 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:51.355884    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:53.390944 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:53.390909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:55.266755 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:55.266711    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:55.531045 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:55.531009    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:56.043818 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:56.043779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:56.525310 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:56.525210    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:56.732668 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:56.732630    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:57.264715 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:57.264672    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:59.362596 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:57:59.362558    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:00.412746 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:00.412700    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:00.625654 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:00.625618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:01.542907 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:01.542871    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:02.079723 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:02.079685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:02.360146 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:02.359999    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:03.115351 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.115308    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 04:58:03.121252 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.121220    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.141754 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.141719    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 04:58:03.149030 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.149000    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-829ac7\"/\"kube-root-ca.crt\""
Oct 31 04:58:03.149398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.149090    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-829ac7\"/\"default-dockercfg-m8bh5\""
Oct 31 04:58:03.149398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.149279    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-829ac7\"/\"test-model-829ac7-kserve-self-signed-certs\""
Oct 31 04:58:03.149398 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.149305    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-829ac7\"/\"openshift-service-ca.crt\""
Oct 31 04:58:03.208106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208069    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.208313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208237    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrp6\" (UniqueName: \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.208313 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208288    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.208411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208334    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.208411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208365    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.208411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.208397    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.267717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.267681    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:03.310486 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310434    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.310760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310537    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.310760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310577    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.310760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310621    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.310760 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310742    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.311017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310874    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrp6\" (UniqueName: \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.311017 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.310919    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.311141 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.311013    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.311322 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.311296    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.313580 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.313556    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.313978 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.313956    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.368148 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.368064    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrp6\" (UniqueName: \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\") pod \"test-model-829ac7-kserve-5c88dfc4f5-t42vg\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") " pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.447513 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.447470    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:03.668388 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.668291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:03.783132 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:03.783085    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 04:58:04.775732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:04.775689    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" event={"ID":"24637e83-a5d7-4480-b83d-1c9327565daa","Type":"ContainerStarted","Data":"40eb206a4139f07f8916630e6139442b8c35d253926ca7e7ee68f2f90379acb8"}
Oct 31 04:58:04.775732 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:04.775738    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" event={"ID":"24637e83-a5d7-4480-b83d-1c9327565daa","Type":"ContainerStarted","Data":"e19e6040433c5b7b7bd3acc09a378823799f3bcc9af2332d6add98713e05a9c8"}
Oct 31 04:58:04.818035 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:04.817977    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" podStartSLOduration=1.817963481 podStartE2EDuration="1.817963481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:04.81580789 +0000 UTC m=+3180.848807586" watchObservedRunningTime="2026-10-31 04:58:04.817963481 +0000 UTC m=+3180.850963159"
Oct 31 04:58:05.447952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:05.447917    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:05.451738 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:05.451703    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:07.178454 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:07.178413    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:09.188092 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:09.187382    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:10.760644 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:58:10.760606    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:58:10.966424 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:10.966380    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:12.958899 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:12.958821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:13.447904 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:13.447873    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:13.460764 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:13.460735    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:13.847997 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:13.847964    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 04:58:14.585591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.585554    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 04:58:14.590411 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.590392    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.617588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.617556    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-47e9af-serving-cert\""
Oct 31 04:58:14.623095 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.623061    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 04:58:14.680384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.680349    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.680599 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.680543    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.680671 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.680628    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.680733 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.680715    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbdx\" (UniqueName: \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.780868 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.780820    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:14.781615 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.781592    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.781757 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.781739    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.781830 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.781795    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.781891 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.781876    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbdx\" (UniqueName: \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.782520 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.782500    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.784282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.784263    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.784371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.784290    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.804765 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.804737    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbdx\" (UniqueName: \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\") pod \"maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:14.905256 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:14.905144    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:15.013387 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.013353    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:15.220888 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.220804    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 04:58:15.226215 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:58:15.226184    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac27bec_2640_4894_bc1b_4b673c3bc6dc.slice/crio-c44d0bb9dfb8098b0ae09f536a5a5147a9cf87c3ae016874c408de8424c95c59 WatchSource:0}: Error finding container c44d0bb9dfb8098b0ae09f536a5a5147a9cf87c3ae016874c408de8424c95c59: Status 404 returned error can't find the container with id c44d0bb9dfb8098b0ae09f536a5a5147a9cf87c3ae016874c408de8424c95c59
Oct 31 04:58:15.231450 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.231405    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:15.920425 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.920322    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" event={"ID":"bac27bec-2640-4894-bc1b-4b673c3bc6dc","Type":"ContainerStarted","Data":"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"}
Oct 31 04:58:15.921309 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.921282    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" event={"ID":"bac27bec-2640-4894-bc1b-4b673c3bc6dc","Type":"ContainerStarted","Data":"c44d0bb9dfb8098b0ae09f536a5a5147a9cf87c3ae016874c408de8424c95c59"}
Oct 31 04:58:15.922582 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.922561    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:15.968661 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:15.968563    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" podStartSLOduration=1.561216113 podStartE2EDuration="1.96854159s" totalImagesPullingTime="407.325477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:15.967439003 +0000 UTC m=+3192.000438718" watchObservedRunningTime="2026-10-31 04:58:15.96854159 +0000 UTC m=+3192.001541270"
Oct 31 04:58:16.689650 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:16.689616    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:16.901509 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:16.901464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:18.593359 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:18.593306    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:20.467453 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:20.467410    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:22.302892 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:22.302854    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:22.943391 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:22.943354    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 04:58:24.231187 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:24.230148    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:26.088608 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:26.088570    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:27.894653 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:27.894618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:29.712054 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:29.712015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:31.277345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.277302    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 04:58:31.282591 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.282570    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.309588 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.309552    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c1fe26-serving-cert\""
Oct 31 04:58:31.315202 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.315146    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 04:58:31.337056 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.337014    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.337282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.337097    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpwg\" (UniqueName: \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.337282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.337197    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.337282 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.337275    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.438865 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.438829    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.439067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.438887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpwg\" (UniqueName: \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.439067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.438955    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.439067 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.439029    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.439279 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:58:31.439218    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-serving-cert: secret "maas-api-e2e-inf-b-c1fe26-serving-cert" not found
Oct 31 04:58:31.439338 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:58:31.439284    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls podName:14ec7070-97da-4ec2-9261-fdf77c5a08c0 nodeName:}" failed. No retries permitted until 2026-10-31 04:58:31.939268324 +0000 UTC m=+3207.972267981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls") pod "maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" (UID: "14ec7070-97da-4ec2-9261-fdf77c5a08c0") : secret "maas-api-e2e-inf-b-c1fe26-serving-cert" not found
Oct 31 04:58:31.439717 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.439699    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.441535 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.441506    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.462446 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.462416    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpwg\" (UniqueName: \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.470384 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.470346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:31.700952 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.700913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:31.945850 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.945811    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:31.948437 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:31.948410    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:32.198479 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:32.198441    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:32.541345 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:32.535743    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 04:58:32.583855 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:58:32.583813    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ec7070_97da_4ec2_9261_fdf77c5a08c0.slice/crio-e3fff9417d2e569428ad502a1823a77e756a40cb6c94dcc43089334266102880 WatchSource:0}: Error finding container e3fff9417d2e569428ad502a1823a77e756a40cb6c94dcc43089334266102880: Status 404 returned error can't find the container with id e3fff9417d2e569428ad502a1823a77e756a40cb6c94dcc43089334266102880
Oct 31 04:58:33.045747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:33.045709    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" event={"ID":"14ec7070-97da-4ec2-9261-fdf77c5a08c0","Type":"ContainerStarted","Data":"5993b264bc9df35232bd05c8f9bd450517655e02aab8ffd4efcc9eb5dbc8c609"}
Oct 31 04:58:33.045921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:33.045805    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:33.045921 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:33.045826    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" event={"ID":"14ec7070-97da-4ec2-9261-fdf77c5a08c0","Type":"ContainerStarted","Data":"e3fff9417d2e569428ad502a1823a77e756a40cb6c94dcc43089334266102880"}
Oct 31 04:58:33.088832 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:33.088771    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" podStartSLOduration=1.812513852 podStartE2EDuration="2.088754029s" totalImagesPullingTime="276.240177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:58:33.086658563 +0000 UTC m=+3209.119658244" watchObservedRunningTime="2026-10-31 04:58:33.088754029 +0000 UTC m=+3209.121753709"
Oct 31 04:58:33.446074 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:33.446033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:35.300851 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:35.300809    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:37.123567 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:37.123529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:38.925178 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:38.925073    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:39.058635 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:39.058605    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 04:58:40.188565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:40.188523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:40.793690 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:58:40.793650    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:58:40.794341 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:40.794300    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:42.617880 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:42.617850    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:44.374016 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:44.373975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:46.089747 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:46.089706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:47.988224 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:47.988179    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:49.712065 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:49.712027    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:51.446941 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:51.446894    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:53.512958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:53.512917    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:55.462958 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:55.462921    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:57.234931 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:57.234790    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:59.025807 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:58:59.025769    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:00.909779 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:00.909739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:02.191371 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:02.191335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:02.405172 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:02.405121    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:02.717204 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:02.716850    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:04.287683 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.287648    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 04:59:04.292094 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.292075    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.306839 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.306815    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-12c0bc-serving-cert\""
Oct 31 04:59:04.327629 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.327595    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 04:59:04.338118 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.338080    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.338343 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.338265    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.338432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.338363    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwx7\" (UniqueName: \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.338432 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.338406    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.439988 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.439952    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwx7\" (UniqueName: \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.440227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.439996    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.440227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.440122    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.440227 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:59:04.440167    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-serving-cert: secret "maas-api-e2e-inf-b-12c0bc-serving-cert" not found
Oct 31 04:59:04.440227 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.440201    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.440481 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:59:04.440231    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls podName:f01e9a3c-c437-484f-a9a4-a1034aaeb9fd nodeName:}" failed. No retries permitted until 2026-10-31 04:59:04.940216226 +0000 UTC m=+3240.973215884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls") pod "maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" (UID: "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd") : secret "maas-api-e2e-inf-b-12c0bc-serving-cert" not found
Oct 31 04:59:04.440857 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.440834    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.442660 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.442635    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.462948 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.462920    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwx7\" (UniqueName: \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.497249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.497209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:04.718655 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.718560    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:04.948080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.948027    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.950510 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.950481    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") pod \"maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:04.961707 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:04.961671    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:05.207377 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:05.207340    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:05.553536 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:05.553506    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 04:59:05.555715 ip-10-0-140-110 kubenswrapper[6931]: W1031 04:59:05.555683    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01e9a3c_c437_484f_a9a4_a1034aaeb9fd.slice/crio-84d116d2ebccf33400808d98a198bd9e2148838133786faee318dbfe1a3be17d WatchSource:0}: Error finding container 84d116d2ebccf33400808d98a198bd9e2148838133786faee318dbfe1a3be17d: Status 404 returned error can't find the container with id 84d116d2ebccf33400808d98a198bd9e2148838133786faee318dbfe1a3be17d
Oct 31 04:59:06.281879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:06.281847    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" event={"ID":"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd","Type":"ContainerStarted","Data":"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"}
Oct 31 04:59:06.282046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:06.281899    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" event={"ID":"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd","Type":"ContainerStarted","Data":"84d116d2ebccf33400808d98a198bd9e2148838133786faee318dbfe1a3be17d"}
Oct 31 04:59:06.282046 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:06.281935    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:06.327405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:06.327347    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" podStartSLOduration=1.937206258 podStartE2EDuration="2.327331126s" totalImagesPullingTime="390.124868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:06.323642306 +0000 UTC m=+3242.356642005" watchObservedRunningTime="2026-10-31 04:59:06.327331126 +0000 UTC m=+3242.360330836"
Oct 31 04:59:06.400602 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:06.400567    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:08.107970 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:08.107925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:10.006201 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:10.006094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:10.761250 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:59:10.761213    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:59:11.785603 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:11.785568    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:12.294879 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:12.294851    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 04:59:13.613080 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:13.613041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:15.407632 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:15.407528    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:17.194439 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:17.194389    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:19.012209 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:19.012133    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:20.797171 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:20.797120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:22.660579 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:22.660537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:23.444245 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:23.444204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:24.661149 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:24.661097    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:24.874324 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:24.874274    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:26.623565 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:26.623529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:28.406578 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:28.406535    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:30.117405 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:30.117363    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:32.025812 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:32.025764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:33.793870 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:33.793832    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:35.674911 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:35.674875    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:37.270545 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:37.270492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:37.480106 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:37.480070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:37.690490 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:37.690445    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:39.371817 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:39.371784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:40.761107 ip-10-0-140-110 kubenswrapper[6931]: E1031 04:59:40.761068    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 04:59:41.307887 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:41.307843    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:43.076041 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:43.076000    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:44.703719 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:44.703645    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:44.923136 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:44.923101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:46.883115 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:46.883072    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:47.907975 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:47.907938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:48.739234 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:48.739189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:50.985383 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:50.985349    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:51.198787 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:51.197332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:53.285576 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:53.285544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:55.105693 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:55.105656    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:56.941604 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:56.941562    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:58.819249 ip-10-0-140-110 kubenswrapper[6931]: I1031 04:59:58.818327    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:00.650988 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:00.650898    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:02.367478 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:02.367439    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:02.740980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:02.740897    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:04.761816 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:04.761778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:06.711266 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:06.711233    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:08.625078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:08.625035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:10.534762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:10.534725    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:10.761878 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:00:10.761843    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:00:12.276132 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:12.276084    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:14.137874 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:14.137835    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:14.363897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:14.363855    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:16.112835 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:16.112797    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:17.927204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:17.927167    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:18.631238 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:18.631198    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:19.816996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:19.816954    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:21.836427 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:21.836338    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:23.748501 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:23.748454    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:25.703104 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:25.703065    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:25.914259 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:25.914212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:27.188372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:27.188334    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:27.414075 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:27.414038    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:27.630953 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:27.630909    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:29.408193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:29.408144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:31.199023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:31.198988    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:33.000496 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:33.000458    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:34.718701 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:34.718659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:36.432372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:36.432332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:38.228441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:38.223788    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:40.091933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:40.091894    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:40.761184 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:00:40.761131    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:00:41.802772 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:41.802732    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:43.514364 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:43.514313    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:45.506455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:45.506356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:47.497472 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:47.497428    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:49.324281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:49.324237    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:51.111481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:51.111442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:52.914984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:52.914949    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:54.726555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:54.726497    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:56.518636 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:56.518597    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:00:58.292978 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:00:58.292939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:00.076028 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:00.075985    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:01.818098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:01.818061    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 05:01:01.818900 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:01.818855    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy" containerID="cri-o://2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85" gracePeriod=30
Oct 31 05:01:01.824889 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:01.824852    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:02.485479 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:02.485035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:03.526023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:03.525990    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:05.275237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:05.275204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:07.018013 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.017978    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:07.145202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.145151    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 05:01:07.179936 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.179896    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.179957    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180045    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180083    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180124    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180399 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180270    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180399 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180282    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:07.180510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180397    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:07.180510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180431    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180497    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180545    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\") pod \"d8ad350c-6196-420e-9d8b-2210b5ff7152\" (UID: \"d8ad350c-6196-420e-9d8b-2210b5ff7152\") "
Oct 31 05:01:07.180674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180549    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:07.180810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180781    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:07.180907 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.180878    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:01:07.181307 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.181286    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.181405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.181314    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.181405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.181332    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.181405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.181348    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ad350c-6196-420e-9d8b-2210b5ff7152-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.181405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.181362    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.182515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.182493    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "kube-api-access-bm4hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:01:07.182599 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.182571    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:01:07.182779 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.182762    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:07.182843 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.182780    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token" pod "d8ad350c-6196-420e-9d8b-2210b5ff7152" (UID: "d8ad350c-6196-420e-9d8b-2210b5ff7152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:01:07.274603 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.274567    6931 generic.go:356] "Generic (PLEG): container finished" podID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerID="2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85" exitCode=0
Oct 31 05:01:07.274808 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.274611    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"
Oct 31 05:01:07.274808 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.274650    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" event={"ID":"d8ad350c-6196-420e-9d8b-2210b5ff7152","Type":"ContainerDied","Data":"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"}
Oct 31 05:01:07.274808 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.274708    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx" event={"ID":"d8ad350c-6196-420e-9d8b-2210b5ff7152","Type":"ContainerDied","Data":"2a492d88e7eba8ac07209bcc17033ec3b770d34a4c53ebb498a6e1a6849865b6"}
Oct 31 05:01:07.274808 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.274739    6931 scope.go:122] "RemoveContainer" containerID="2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"
Oct 31 05:01:07.283074 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.283051    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.283193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.283077    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.283193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.283087    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4hm\" (UniqueName: \"kubernetes.io/projected/d8ad350c-6196-420e-9d8b-2210b5ff7152-kube-api-access-bm4hm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.283193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.283098    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ad350c-6196-420e-9d8b-2210b5ff7152-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:07.286690 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.286670    6931 scope.go:122] "RemoveContainer" containerID="2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"
Oct 31 05:01:07.287016 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:01:07.286992    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85\": container with ID starting with 2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85 not found: ID does not exist" containerID="2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"
Oct 31 05:01:07.287127 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.287021    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85"} err="failed to get container status \"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85\": rpc error: code = NotFound desc = could not find container \"2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85\": container with ID starting with 2dbd0f0f6aeb32b55bdcd0aaee6449d69c0b16250a44a20058d2038b5048dc85 not found: ID does not exist"
Oct 31 05:01:07.331074 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.331034    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 05:01:07.334779 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:07.334747    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j7hwx"]
Oct 31 05:01:08.124151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:08.124117    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" path="/var/lib/kubelet/pods/d8ad350c-6196-420e-9d8b-2210b5ff7152/volumes"
Oct 31 05:01:08.187149 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:08.187112    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 05:01:08.187364 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:08.187341    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy" containerID="cri-o://631cdf3d5ed9ad348cfe082751199772c28f3c2ab917ff36351ca9fd79abb941" gracePeriod=30
Oct 31 05:01:08.881849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:08.881764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:10.640032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:10.639999    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:10.761299 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:01:10.761263    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:01:12.418672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:12.418634    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:13.320399 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.320360    6931 generic.go:356] "Generic (PLEG): container finished" podID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerID="631cdf3d5ed9ad348cfe082751199772c28f3c2ab917ff36351ca9fd79abb941" exitCode=0
Oct 31 05:01:13.320588 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.320422    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" event={"ID":"df8d7d49-d719-4fed-971b-89f538d0e39c","Type":"ContainerDied","Data":"631cdf3d5ed9ad348cfe082751199772c28f3c2ab917ff36351ca9fd79abb941"}
Oct 31 05:01:13.479032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.479004    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 05:01:13.575340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575304    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575513 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575407    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575513 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575450    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575537    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575585    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575654    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:13.575763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575670    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575704    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575806    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.575923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.575880    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\") pod \"df8d7d49-d719-4fed-971b-89f538d0e39c\" (UID: \"df8d7d49-d719-4fed-971b-89f538d0e39c\") "
Oct 31 05:01:13.576516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.576359    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:01:13.576516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.576438    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:13.576516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.576487    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.576516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.576492    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:13.576985 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.576744    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:13.578241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.578200    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:01:13.578609 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.578582    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "kube-api-access-pqj7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:01:13.578800 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.578775    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:01:13.579049 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.579028    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token" pod "df8d7d49-d719-4fed-971b-89f538d0e39c" (UID: "df8d7d49-d719-4fed-971b-89f538d0e39c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:01:13.678258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678216    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678249    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678259    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8d7d49-d719-4fed-971b-89f538d0e39c-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678268    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqj7f\" (UniqueName: \"kubernetes.io/projected/df8d7d49-d719-4fed-971b-89f538d0e39c-kube-api-access-pqj7f\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678280    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678288    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8d7d49-d719-4fed-971b-89f538d0e39c-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678296    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:13.678525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:13.678306    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8d7d49-d719-4fed-971b-89f538d0e39c-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:01:14.235925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.235888    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:14.328821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.328792    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"
Oct 31 05:01:14.329039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.328828    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm" event={"ID":"df8d7d49-d719-4fed-971b-89f538d0e39c","Type":"ContainerDied","Data":"786581a7dcca690034295be37d17460d1c9092ff2dd2cd7f819d702ad7904492"}
Oct 31 05:01:14.329039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.328900    6931 scope.go:122] "RemoveContainer" containerID="631cdf3d5ed9ad348cfe082751199772c28f3c2ab917ff36351ca9fd79abb941"
Oct 31 05:01:14.393839 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.393753    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 05:01:14.406333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:14.406296    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-439989-openshift-default-8448c6ff99-g9wnm"]
Oct 31 05:01:16.037836 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:16.037784    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:16.125199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:16.125150    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" path="/var/lib/kubelet/pods/df8d7d49-d719-4fed-971b-89f538d0e39c/volumes"
Oct 31 05:01:17.993076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:17.993027    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:19.762341 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:19.762301    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:21.689754 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:21.689718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:23.436929 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:23.436816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:25.201729 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:25.201685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:26.994307 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:26.994267    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:28.717864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:28.717823    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:30.446744 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:30.446714    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:32.243776 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:32.243723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:34.124743 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:34.124695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:36.029478 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:36.029438    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:37.803415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:37.803381    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:39.529635 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:39.529597    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:40.760033 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:01:40.759996    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:01:41.417444 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:41.417401    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:43.216479 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:43.216437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:43.433854 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:43.433814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:45.012197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:45.012130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:46.740345 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:46.740309    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:48.521849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:48.521807    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:50.400492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:50.400442    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:52.277062 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:52.277025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:54.118665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:54.118626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:55.850132 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:55.850094    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:57.626221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:57.626148    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:01:59.500535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:01:59.500491    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:01.232070 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:01.232036    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:02.616196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:02.615590    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:03.115090 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:03.115047    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:04.920496 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:04.920452    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:06.775580 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:06.775542    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:08.592924 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:08.592886    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:10.388781 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:10.388738    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:10.760465 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:02:10.760381    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:02:12.283735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:12.283691    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:12.505188 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:12.505120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:14.212625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:14.212515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:16.093543 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:16.093500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:17.908982 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:17.908940    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:19.707917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:19.707878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:21.538221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:21.538172    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:23.269926 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:23.269881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:25.051198 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:25.051117    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:26.923337 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:26.923295    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:28.673190 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:28.673112    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:30.480974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:30.480933    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:32.220072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:32.220039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:34.003087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:34.003050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:35.799127 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:35.799086    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:37.605903 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:37.605867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:39.402126 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:39.402085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:40.760107 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:02:40.760069    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:02:41.141447 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:41.141373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:42.192392 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:42.192346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:43.112537 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:43.112502    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:44.900835 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:44.900794    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:46.694325 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:46.694291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:48.518973 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:48.518928    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:50.287415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:50.287373    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:52.025977 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:52.025934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:53.850199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:53.850144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:55.691328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:55.691292    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:57.423604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:57.423567    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:02:59.215489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:02:59.215382    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:01.016136 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:01.016104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:02.777063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:02.777019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:02.982895 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:02.982860    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:04.029652 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:04.029607    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 05:03:04.188583 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:04.188544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:04.630569 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:04.630537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:04.838648 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:04.838611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:06.338644 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:06.338597    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:08.203849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:08.203816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:08.567964 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:08.567929    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:08.782375 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:08.782330    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:09.925421 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:09.925393    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:10.413208 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:10.413138    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:10.432695 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:10.432661    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 05:03:10.469448 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:10.469403    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 05:03:10.681032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:10.677495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:10.761500 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:03:10.761466    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:03:10.896197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:10.896139    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:11.866394 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:11.866355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:13.604134 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:13.604092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:15.414314 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:15.414279    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:15.432981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:15.432946    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing" containerID="cri-o://93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa" gracePeriod=25
Oct 31 05:03:15.470814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:15.470779    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing" containerID="cri-o://8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b" gracePeriod=25
Oct 31 05:03:16.195197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:16.195144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:17.127749 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:17.127713    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:18.930764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:18.930724    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:19.053012 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.052975    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" podUID="24637e83-a5d7-4480-b83d-1c9327565daa" containerName="main" containerID="cri-o://40eb206a4139f07f8916630e6139442b8c35d253926ca7e7ee68f2f90379acb8" gracePeriod=45
Oct 31 05:03:19.339505 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.339468    6931 generic.go:356] "Generic (PLEG): container finished" podID="24637e83-a5d7-4480-b83d-1c9327565daa" containerID="40eb206a4139f07f8916630e6139442b8c35d253926ca7e7ee68f2f90379acb8" exitCode=0
Oct 31 05:03:19.339684 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.339557    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" event={"ID":"24637e83-a5d7-4480-b83d-1c9327565daa","Type":"ContainerDied","Data":"40eb206a4139f07f8916630e6139442b8c35d253926ca7e7ee68f2f90379acb8"}
Oct 31 05:03:19.339684 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.339598    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg" event={"ID":"24637e83-a5d7-4480-b83d-1c9327565daa","Type":"ContainerDied","Data":"e19e6040433c5b7b7bd3acc09a378823799f3bcc9af2332d6add98713e05a9c8"}
Oct 31 05:03:19.339684 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.339613    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19e6040433c5b7b7bd3acc09a378823799f3bcc9af2332d6add98713e05a9c8"
Oct 31 05:03:19.353035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.353008    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 05:03:19.496768 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.496673    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\" (UniqueName: \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.496768 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.496733    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.497035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.496811    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.497035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.496899    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.497035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.496952    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\" (UniqueName: \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.497035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497001    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\") pod \"24637e83-a5d7-4480-b83d-1c9327565daa\" (UID: \"24637e83-a5d7-4480-b83d-1c9327565daa\") "
Oct 31 05:03:19.497252 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497034    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:03:19.497252 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497209    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:03:19.497364 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497332    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:03:19.497556 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497535    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:19.497656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497560    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:19.497656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.497569    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:19.499106 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.499083    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:03:19.499218 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.499196    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:03:19.499267 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.499218    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6" pod "24637e83-a5d7-4480-b83d-1c9327565daa" (UID: "24637e83-a5d7-4480-b83d-1c9327565daa"). InnerVolumeSpecName "kube-api-access-wqrp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:03:19.598408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.598369    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24637e83-a5d7-4480-b83d-1c9327565daa-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:19.598408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.598402    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrp6\" (UniqueName: \"kubernetes.io/projected/24637e83-a5d7-4480-b83d-1c9327565daa-kube-api-access-wqrp6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:19.598408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:19.598412    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24637e83-a5d7-4480-b83d-1c9327565daa-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:20.347091 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:20.347057    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"
Oct 31 05:03:20.398960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:20.398869    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 05:03:20.410689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:20.410646    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-829ac7/test-model-829ac7-kserve-5c88dfc4f5-t42vg"]
Oct 31 05:03:20.809209 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:20.809149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:22.125509 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:22.125475    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24637e83-a5d7-4480-b83d-1c9327565daa" path="/var/lib/kubelet/pods/24637e83-a5d7-4480-b83d-1c9327565daa/volumes"
Oct 31 05:03:22.571521 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:22.571480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:23.485653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:23.485614    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:03:23.486103 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:23.485615    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:03:24.341397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:24.341354    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:26.288139 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:26.288104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:28.119514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:28.119481    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:29.815731 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:29.815694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:31.590651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:31.590615    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:33.471672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:33.471632    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:33.485385 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:33.485357    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:03:33.485529 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:33.485360    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:03:35.182572 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.182530    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:35.755796 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.755770    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 05:03:35.818691 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.818662    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 05:03:35.926062 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926019    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\") pod \"23c539ae-d39a-42c8-b1e3-2754023442fc\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") "
Oct 31 05:03:35.926280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926086    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\") pod \"d7e60e01-bce0-4ea9-aad3-f888dda53189\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") "
Oct 31 05:03:35.926280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926108    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\" (UniqueName: \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\") pod \"d7e60e01-bce0-4ea9-aad3-f888dda53189\" (UID: \"d7e60e01-bce0-4ea9-aad3-f888dda53189\") "
Oct 31 05:03:35.926280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926130    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\" (UniqueName: \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\") pod \"23c539ae-d39a-42c8-b1e3-2754023442fc\" (UID: \"23c539ae-d39a-42c8-b1e3-2754023442fc\") "
Oct 31 05:03:35.926501 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926479    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume" pod "23c539ae-d39a-42c8-b1e3-2754023442fc" (UID: "23c539ae-d39a-42c8-b1e3-2754023442fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:03:35.926589 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926557    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume" pod "d7e60e01-bce0-4ea9-aad3-f888dda53189" (UID: "d7e60e01-bce0-4ea9-aad3-f888dda53189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:03:35.926707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926688    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c539ae-d39a-42c8-b1e3-2754023442fc-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:35.926707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.926704    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e60e01-bce0-4ea9-aad3-f888dda53189-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:35.928502 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.928472    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8" pod "d7e60e01-bce0-4ea9-aad3-f888dda53189" (UID: "d7e60e01-bce0-4ea9-aad3-f888dda53189"). InnerVolumeSpecName "kube-api-access-zcwj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:03:35.928641 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:35.928536    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z" pod "23c539ae-d39a-42c8-b1e3-2754023442fc" (UID: "23c539ae-d39a-42c8-b1e3-2754023442fc"). InnerVolumeSpecName "kube-api-access-87g7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:03:36.028023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.027929    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwj8\" (UniqueName: \"kubernetes.io/projected/d7e60e01-bce0-4ea9-aad3-f888dda53189-kube-api-access-zcwj8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:36.028023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.027963    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87g7z\" (UniqueName: \"kubernetes.io/projected/23c539ae-d39a-42c8-b1e3-2754023442fc-kube-api-access-87g7z\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:36.466620 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.466579    6931 generic.go:356] "Generic (PLEG): container finished" podID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerID="8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b" exitCode=0
Oct 31 05:03:36.467078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.466623    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"
Oct 31 05:03:36.467078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.466656    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" event={"ID":"d7e60e01-bce0-4ea9-aad3-f888dda53189","Type":"ContainerDied","Data":"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"}
Oct 31 05:03:36.467078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.466716    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj" event={"ID":"d7e60e01-bce0-4ea9-aad3-f888dda53189","Type":"ContainerDied","Data":"5d24df986aa8f46e802b59b378deaebac6030fe41a516f64dc9655db7a38a677"}
Oct 31 05:03:36.467078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.466740    6931 scope.go:122] "RemoveContainer" containerID="8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"
Oct 31 05:03:36.468772 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.468748    6931 generic.go:356] "Generic (PLEG): container finished" podID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerID="93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa" exitCode=0
Oct 31 05:03:36.468886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.468800    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"
Oct 31 05:03:36.468886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.468814    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" event={"ID":"23c539ae-d39a-42c8-b1e3-2754023442fc","Type":"ContainerDied","Data":"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"}
Oct 31 05:03:36.468886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.468855    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs" event={"ID":"23c539ae-d39a-42c8-b1e3-2754023442fc","Type":"ContainerDied","Data":"fbba12a0f2a30c5b269190ab74c983a93d91328688325dae2fcad880e24c9ee4"}
Oct 31 05:03:36.480445 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.480428    6931 scope.go:122] "RemoveContainer" containerID="8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"
Oct 31 05:03:36.480840 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:03:36.480813    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b\": container with ID starting with 8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b not found: ID does not exist" containerID="8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"
Oct 31 05:03:36.480947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.480849    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b"} err="failed to get container status \"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b\": rpc error: code = NotFound desc = could not find container \"8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b\": container with ID starting with 8978cd1e428e1acacb266792b7e3781852c9367455c18880f0037452173a562b not found: ID does not exist"
Oct 31 05:03:36.480947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.480872    6931 scope.go:122] "RemoveContainer" containerID="93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"
Oct 31 05:03:36.491247 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.491227    6931 scope.go:122] "RemoveContainer" containerID="93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"
Oct 31 05:03:36.491524 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:03:36.491506    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa\": container with ID starting with 93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa not found: ID does not exist" containerID="93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"
Oct 31 05:03:36.491583 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.491534    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa"} err="failed to get container status \"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa\": rpc error: code = NotFound desc = could not find container \"93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa\": container with ID starting with 93969edec44ce7ea1a937ce7a69bec525439f01c8e72c3f9fa95860b417c57fa not found: ID does not exist"
Oct 31 05:03:36.518789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.518735    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 05:03:36.533026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.532990    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-670795-788dc77bff-j9zfj"]
Oct 31 05:03:36.557519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.557473    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 05:03:36.561025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.560984    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-670795-8c86c56f7-d5jqs"]
Oct 31 05:03:36.917563 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:36.917524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:38.124382 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:38.124344    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" path="/var/lib/kubelet/pods/23c539ae-d39a-42c8-b1e3-2754023442fc/volumes"
Oct 31 05:03:38.124825 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:38.124810    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" path="/var/lib/kubelet/pods/d7e60e01-bce0-4ea9-aad3-f888dda53189/volumes"
Oct 31 05:03:38.636759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:38.636716    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:40.423260 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:40.423217    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:40.761187 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:03:40.761069    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:03:42.399901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:42.399839    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:44.200503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:44.200411    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:46.207725 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:46.207689    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:48.112220 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:48.112149    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:49.901607 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:49.901563    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:51.640865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:51.640831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:53.494898 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:53.494855    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:55.394295 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:55.394255    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:57.113607 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:57.113564    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:58.500526 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.500479    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 05:03:58.503258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.500748    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" podUID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" containerName="maas-api" containerID="cri-o://72c34809bd9866369b4cbafad6750f56e8eaad57af5f549a85b299fcaeb9a049" gracePeriod=30
Oct 31 05:03:58.627744 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.627710    6931 generic.go:356] "Generic (PLEG): container finished" podID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" containerID="72c34809bd9866369b4cbafad6750f56e8eaad57af5f549a85b299fcaeb9a049" exitCode=0
Oct 31 05:03:58.627934 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.627768    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" event={"ID":"8278703c-e2f4-48f7-b7d0-7842f5f65a29","Type":"ContainerDied","Data":"72c34809bd9866369b4cbafad6750f56e8eaad57af5f549a85b299fcaeb9a049"}
Oct 31 05:03:58.672200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.671519    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:58.877181 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.876323    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 05:03:58.945200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.935656    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\") pod \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") "
Oct 31 05:03:58.945200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.935707    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\") pod \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") "
Oct 31 05:03:58.945200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.935806    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\") pod \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") "
Oct 31 05:03:58.945200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.935992    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\" (UniqueName: \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\") pod \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\" (UID: \"8278703c-e2f4-48f7-b7d0-7842f5f65a29\") "
Oct 31 05:03:58.968289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.957949    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca" pod "8278703c-e2f4-48f7-b7d0-7842f5f65a29" (UID: "8278703c-e2f4-48f7-b7d0-7842f5f65a29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:03:58.968289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.958693    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47" pod "8278703c-e2f4-48f7-b7d0-7842f5f65a29" (UID: "8278703c-e2f4-48f7-b7d0-7842f5f65a29"). InnerVolumeSpecName "kube-api-access-p5r47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:03:58.983194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.975290    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls" pod "8278703c-e2f4-48f7-b7d0-7842f5f65a29" (UID: "8278703c-e2f4-48f7-b7d0-7842f5f65a29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:03:58.989178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:58.984368    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs" pod "8278703c-e2f4-48f7-b7d0-7842f5f65a29" (UID: "8278703c-e2f4-48f7-b7d0-7842f5f65a29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:03:59.039180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.038614    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5r47\" (UniqueName: \"kubernetes.io/projected/8278703c-e2f4-48f7-b7d0-7842f5f65a29-kube-api-access-p5r47\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:59.039180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.038658    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:59.039180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.038674    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8278703c-e2f4-48f7-b7d0-7842f5f65a29-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:59.039180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.038690    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8278703c-e2f4-48f7-b7d0-7842f5f65a29-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:03:59.500714 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.500677    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:03:59.635866 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.635835    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"
Oct 31 05:03:59.636074 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.635877    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs" event={"ID":"8278703c-e2f4-48f7-b7d0-7842f5f65a29","Type":"ContainerDied","Data":"3e56436e315234168f2f6e20897847af22e2a18ff6b8ef7cd9470c30f48d7144"}
Oct 31 05:03:59.636074 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.635948    6931 scope.go:122] "RemoveContainer" containerID="72c34809bd9866369b4cbafad6750f56e8eaad57af5f549a85b299fcaeb9a049"
Oct 31 05:03:59.686975 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.686943    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 05:03:59.698383 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:03:59.698354    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-670795-5747fd846d-ss9zs"]
Oct 31 05:04:00.124346 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:00.124311    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" path="/var/lib/kubelet/pods/8278703c-e2f4-48f7-b7d0-7842f5f65a29/volumes"
Oct 31 05:04:01.328858 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:01.328824    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:02.880135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:02.880102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:03.095006 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:03.094965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:04.840028 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:04.839989    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:05.390654 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:05.390621    6931 scope.go:122] "RemoveContainer" containerID="40eb206a4139f07f8916630e6139442b8c35d253926ca7e7ee68f2f90379acb8"
Oct 31 05:04:06.640144 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:06.640103    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:06.859565 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:06.859534    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:07.497196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:07.497126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:07.915217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:07.915140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:09.719563 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:09.719527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:10.452552 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:10.452516    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:10.760802 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:04:10.760698    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:04:12.282248 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:12.282211    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:14.018677 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:14.018638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:14.607012 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:14.606977    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 05:04:14.851748 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:14.851708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:15.061736 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:15.061690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:15.532342 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:15.532299    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:16.086902 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:16.086863    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:17.816465 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:17.816425    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:19.517214 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:19.517170    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:19.608035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:19.607965    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing" containerID="cri-o://28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900" gracePeriod=25
Oct 31 05:04:21.301223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:21.301146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:21.553452 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:21.553370    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:23.087944 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:23.087905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:23.417174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:23.417054    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:24.994347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:24.994300    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:26.559801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:26.559747    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:04:26.824882 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:26.824798    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:28.608825 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:28.608782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:30.326771 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:30.326731    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:32.109403 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:32.109364    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:33.831099 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:33.831056    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:34.047945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:34.047904    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:35.696964 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:35.696918    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:36.560308 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:36.560265    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:04:37.494371 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:37.494324    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:39.202520 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.202487    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:39.888178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.888127    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 05:04:39.936197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.936140    6931 generic.go:356] "Generic (PLEG): container finished" podID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerID="28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900" exitCode=0
Oct 31 05:04:39.936387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.936204    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"
Oct 31 05:04:39.936387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.936202    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" event={"ID":"7c3d97e2-092e-4a1b-8509-94d92225e1b6","Type":"ContainerDied","Data":"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"}
Oct 31 05:04:39.936387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.936286    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554" event={"ID":"7c3d97e2-092e-4a1b-8509-94d92225e1b6","Type":"ContainerDied","Data":"11899a19e11925f999ed59ffe92767a25954c577dcdd175ed0feaf7a4d194b36"}
Oct 31 05:04:39.936387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.936313    6931 scope.go:122] "RemoveContainer" containerID="28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"
Oct 31 05:04:39.949426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.949399    6931 scope.go:122] "RemoveContainer" containerID="28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"
Oct 31 05:04:39.949787 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:04:39.949759    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900\": container with ID starting with 28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900 not found: ID does not exist" containerID="28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"
Oct 31 05:04:39.949884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:39.949799    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900"} err="failed to get container status \"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900\": rpc error: code = NotFound desc = could not find container \"28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900\": container with ID starting with 28b48eb475ad4910c5ea03982f7aee71e1d7724c79aae34a57c89bd9beef1900 not found: ID does not exist"
Oct 31 05:04:40.015965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.015933    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\" (UniqueName: \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\") pod \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") "
Oct 31 05:04:40.016191 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.016100    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\") pod \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\" (UID: \"7c3d97e2-092e-4a1b-8509-94d92225e1b6\") "
Oct 31 05:04:40.016487 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.016464    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume" pod "7c3d97e2-092e-4a1b-8509-94d92225e1b6" (UID: "7c3d97e2-092e-4a1b-8509-94d92225e1b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:04:40.018273 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.018250    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt" pod "7c3d97e2-092e-4a1b-8509-94d92225e1b6" (UID: "7c3d97e2-092e-4a1b-8509-94d92225e1b6"). InnerVolumeSpecName "kube-api-access-nwtdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:04:40.117991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.117901    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3d97e2-092e-4a1b-8509-94d92225e1b6-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:04:40.117991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.117933    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtdt\" (UniqueName: \"kubernetes.io/projected/7c3d97e2-092e-4a1b-8509-94d92225e1b6-kube-api-access-nwtdt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:04:40.276182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.276119    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 05:04:40.279802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:40.279773    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-829ac7-657c795bc9-t6554"]
Oct 31 05:04:40.760672 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:04:40.760635    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:04:41.110436 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:41.110397    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:42.124123 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:42.124089    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" path="/var/lib/kubelet/pods/7c3d97e2-092e-4a1b-8509-94d92225e1b6/volumes"
Oct 31 05:04:42.843386 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:42.843297    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:44.718961 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:44.718925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:45.291743 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:45.291702    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:45.780064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:45.780029    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:46.520797 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:46.520759    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:48.396199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:48.396120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:50.194423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:50.194377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:51.370363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:51.370313    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 05:04:51.400174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:51.400126    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 05:04:51.610693 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:51.610657    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:51.818488 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:51.818454    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:52.185764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:52.185688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:52.393770 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:52.393737    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:54.204056 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:54.204019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:56.014034 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:56.013996    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:56.371705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:56.371656    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing" containerID="cri-o://531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805" gracePeriod=25
Oct 31 05:04:56.400465 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:56.400420    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing" containerID="cri-o://5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4" gracePeriod=25
Oct 31 05:04:57.815636 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:57.815593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:58.184304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:58.184201    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:04:59.718329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:04:59.718293    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:01.478635 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:01.478597    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:02.998041 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:02.998001    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:03.224931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:03.224895    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:03.724186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:03.724119    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:05:03.724341 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:03.724122    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:05:05.024235 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:05.024199    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:06.768742 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:06.768705    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:08.500031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:08.499987    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:09.872547 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:09.872501    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 05:05:09.873078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:09.872768    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" podUID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" containerName="maas-api" containerID="cri-o://4f29953be3add03a4dd0c45a8982e5759579b2f0fda52ae66dacd4c3cd9095cd" gracePeriod=30
Oct 31 05:05:10.042361 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.042315    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:10.206457 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.206334    6931 generic.go:356] "Generic (PLEG): container finished" podID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" containerID="4f29953be3add03a4dd0c45a8982e5759579b2f0fda52ae66dacd4c3cd9095cd" exitCode=0
Oct 31 05:05:10.207592 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.207530    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" event={"ID":"dda4ea64-5dd6-421e-bfc5-e3b290990ca6","Type":"ContainerDied","Data":"4f29953be3add03a4dd0c45a8982e5759579b2f0fda52ae66dacd4c3cd9095cd"}
Oct 31 05:05:10.273866 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.273835    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 05:05:10.391073 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.391029    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\") pod \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") "
Oct 31 05:05:10.391419 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.391398    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\") pod \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") "
Oct 31 05:05:10.391608 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.391584    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\") pod \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") "
Oct 31 05:05:10.391868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.391852    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\" (UniqueName: \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\") pod \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\" (UID: \"dda4ea64-5dd6-421e-bfc5-e3b290990ca6\") "
Oct 31 05:05:10.393400 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.392390    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca" pod "dda4ea64-5dd6-421e-bfc5-e3b290990ca6" (UID: "dda4ea64-5dd6-421e-bfc5-e3b290990ca6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:10.394995 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.394971    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs" pod "dda4ea64-5dd6-421e-bfc5-e3b290990ca6" (UID: "dda4ea64-5dd6-421e-bfc5-e3b290990ca6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:10.397748 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.397704    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg" pod "dda4ea64-5dd6-421e-bfc5-e3b290990ca6" (UID: "dda4ea64-5dd6-421e-bfc5-e3b290990ca6"). InnerVolumeSpecName "kube-api-access-swtbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:10.399593 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.399570    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls" pod "dda4ea64-5dd6-421e-bfc5-e3b290990ca6" (UID: "dda4ea64-5dd6-421e-bfc5-e3b290990ca6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:10.494897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.494793    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtbg\" (UniqueName: \"kubernetes.io/projected/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-kube-api-access-swtbg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:10.494897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.494835    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:10.494897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.494849    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:10.494897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.494861    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda4ea64-5dd6-421e-bfc5-e3b290990ca6-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:10.760874 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:10.760794    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:05:10.996815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:10.996771    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:11.215365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:11.215326    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv" event={"ID":"dda4ea64-5dd6-421e-bfc5-e3b290990ca6","Type":"ContainerDied","Data":"290ead6ff9932d3a06daa90651f8daa530cd8018456fc1f7f3a8015b66699530"}
Oct 31 05:05:11.215365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:11.215359    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"
Oct 31 05:05:11.215617 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:11.215387    6931 scope.go:122] "RemoveContainer" containerID="4f29953be3add03a4dd0c45a8982e5759579b2f0fda52ae66dacd4c3cd9095cd"
Oct 31 05:05:11.268051 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:11.268017    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 05:05:11.279797 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:11.279759    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-829ac7-64868d9765-5rxnv"]
Oct 31 05:05:12.124028 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:12.123994    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" path="/var/lib/kubelet/pods/dda4ea64-5dd6-421e-bfc5-e3b290990ca6/volumes"
Oct 31 05:05:12.899299 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:12.899255    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:13.724257 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:13.724220    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:05:13.724676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:13.724227    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:05:13.853263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:13.853229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:14.068549 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:14.068514    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:14.671591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:14.671543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:14.888775 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:14.888724    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:15.098613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:15.098539    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:15.931871 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:15.931833    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:16.153936 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.153896    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:16.658290 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.658116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:16.697482 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.697453    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 05:05:16.777314 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.777279    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\" (UniqueName: \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\") pod \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") "
Oct 31 05:05:16.777499 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.777477    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\") pod \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\" (UID: \"1fc6db11-c632-4c17-93b1-ac9c8e2cc246\") "
Oct 31 05:05:16.778719 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.778675    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume" pod "1fc6db11-c632-4c17-93b1-ac9c8e2cc246" (UID: "1fc6db11-c632-4c17-93b1-ac9c8e2cc246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:16.780756 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.780726    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq" pod "1fc6db11-c632-4c17-93b1-ac9c8e2cc246" (UID: "1fc6db11-c632-4c17-93b1-ac9c8e2cc246"). InnerVolumeSpecName "kube-api-access-7wspq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:16.784869 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.784843    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 05:05:16.878614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.878561    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\") pod \"56f11c6a-23af-4678-932c-1aa7c4b85242\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") "
Oct 31 05:05:16.878797 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.878748    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\" (UniqueName: \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\") pod \"56f11c6a-23af-4678-932c-1aa7c4b85242\" (UID: \"56f11c6a-23af-4678-932c-1aa7c4b85242\") "
Oct 31 05:05:16.879040 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.879018    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume" pod "56f11c6a-23af-4678-932c-1aa7c4b85242" (UID: "56f11c6a-23af-4678-932c-1aa7c4b85242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:16.879283 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.879261    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f11c6a-23af-4678-932c-1aa7c4b85242-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:16.879283 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.879282    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:16.879433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.879292    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wspq\" (UniqueName: \"kubernetes.io/projected/1fc6db11-c632-4c17-93b1-ac9c8e2cc246-kube-api-access-7wspq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:16.880782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.880764    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq" pod "56f11c6a-23af-4678-932c-1aa7c4b85242" (UID: "56f11c6a-23af-4678-932c-1aa7c4b85242"). InnerVolumeSpecName "kube-api-access-6cxnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:16.980828 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:16.980742    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxnq\" (UniqueName: \"kubernetes.io/projected/56f11c6a-23af-4678-932c-1aa7c4b85242-kube-api-access-6cxnq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:17.257653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.257569    6931 generic.go:356] "Generic (PLEG): container finished" podID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerID="531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805" exitCode=0
Oct 31 05:05:17.257653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.257621    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"
Oct 31 05:05:17.257865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.257663    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" event={"ID":"1fc6db11-c632-4c17-93b1-ac9c8e2cc246","Type":"ContainerDied","Data":"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"}
Oct 31 05:05:17.257865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.257713    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm" event={"ID":"1fc6db11-c632-4c17-93b1-ac9c8e2cc246","Type":"ContainerDied","Data":"41eb9d2b43d3acf158a589bf165618524c3807dfb54c1e135b85d15421faa903"}
Oct 31 05:05:17.257865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.257742    6931 scope.go:122] "RemoveContainer" containerID="531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"
Oct 31 05:05:17.259196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.259169    6931 generic.go:356] "Generic (PLEG): container finished" podID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerID="5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4" exitCode=0
Oct 31 05:05:17.259292 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.259209    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"
Oct 31 05:05:17.259292 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.259245    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" event={"ID":"56f11c6a-23af-4678-932c-1aa7c4b85242","Type":"ContainerDied","Data":"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"}
Oct 31 05:05:17.259623 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.259306    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb" event={"ID":"56f11c6a-23af-4678-932c-1aa7c4b85242","Type":"ContainerDied","Data":"5ca2b25bd7e99afa844f8d1c38d148118d76f397b54349cb4b3edce127f8b69b"}
Oct 31 05:05:17.274817 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.274792    6931 scope.go:122] "RemoveContainer" containerID="531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"
Oct 31 05:05:17.275105 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:17.275082    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805\": container with ID starting with 531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805 not found: ID does not exist" containerID="531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"
Oct 31 05:05:17.275237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.275117    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805"} err="failed to get container status \"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805\": rpc error: code = NotFound desc = could not find container \"531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805\": container with ID starting with 531b05388885e8a0a64de34807c4b1c57bc3197f6ed97e3cdb883edbbffa4805 not found: ID does not exist"
Oct 31 05:05:17.275237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.275139    6931 scope.go:122] "RemoveContainer" containerID="5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"
Oct 31 05:05:17.289007 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.288980    6931 scope.go:122] "RemoveContainer" containerID="5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"
Oct 31 05:05:17.289397 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:17.289376    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4\": container with ID starting with 5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4 not found: ID does not exist" containerID="5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"
Oct 31 05:05:17.289477 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.289405    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4"} err="failed to get container status \"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4\": rpc error: code = NotFound desc = could not find container \"5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4\": container with ID starting with 5e1089845ef3ecdd12a2df2dd666e9de04c7c873e1edb1af91a65be8484dc9d4 not found: ID does not exist"
Oct 31 05:05:17.311048 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.311012    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 05:05:17.323185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.323141    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-171edd-766c966749-c55nm"]
Oct 31 05:05:17.367577 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.367538    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 05:05:17.371202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.371146    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-171edd-78998dcc55-2v6bb"]
Oct 31 05:05:17.537320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.537283    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 05:05:17.537522 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:17.537498    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy" containerID="cri-o://22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058" gracePeriod=30
Oct 31 05:05:18.123523 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:18.123486    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" path="/var/lib/kubelet/pods/1fc6db11-c632-4c17-93b1-ac9c8e2cc246/volumes"
Oct 31 05:05:18.123922 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:18.123891    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" path="/var/lib/kubelet/pods/56f11c6a-23af-4678-932c-1aa7c4b85242/volumes"
Oct 31 05:05:18.501384 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:18.501273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:20.315012 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:20.314965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:22.199817 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.199774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:22.484221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.483636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:22.827875 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.827845    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 05:05:22.874072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874034    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874316 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874091    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874316 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874141    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874316 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874238    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874498 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874317    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874498 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874373    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874498 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874447    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874632 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874524    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.874632 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874445    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:22.874632 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874457    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:22.874787 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874600    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:22.874787 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.874647    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\") pod \"e717b2a3-78b9-451e-b35a-26987837c05a\" (UID: \"e717b2a3-78b9-451e-b35a-26987837c05a\") "
Oct 31 05:05:22.875140 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.875111    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:22.875844 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.875800    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:22.876101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876080    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.876277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876106    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.876277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876118    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.876277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876131    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.876277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876145    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e717b2a3-78b9-451e-b35a-26987837c05a-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.876928 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876902    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:05:22.877037 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.876947    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:22.877651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.877625    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:22.877741 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.877658    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn" pod "e717b2a3-78b9-451e-b35a-26987837c05a" (UID: "e717b2a3-78b9-451e-b35a-26987837c05a"). InnerVolumeSpecName "kube-api-access-wnjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:22.977549 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.977510    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjzn\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-kube-api-access-wnjzn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.977549 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.977544    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e717b2a3-78b9-451e-b35a-26987837c05a-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.977767 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.977559    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e717b2a3-78b9-451e-b35a-26987837c05a-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:22.977767 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:22.977573    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e717b2a3-78b9-451e-b35a-26987837c05a-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:23.325057 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.325022    6931 generic.go:356] "Generic (PLEG): container finished" podID="e717b2a3-78b9-451e-b35a-26987837c05a" containerID="22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058" exitCode=0
Oct 31 05:05:23.325564 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.325074    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" event={"ID":"e717b2a3-78b9-451e-b35a-26987837c05a","Type":"ContainerDied","Data":"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"}
Oct 31 05:05:23.325564 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.325075    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"
Oct 31 05:05:23.325564 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.325118    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf" event={"ID":"e717b2a3-78b9-451e-b35a-26987837c05a","Type":"ContainerDied","Data":"a6015e1aae8865f096b53b3e0f33092ffaca45f94f07e1d147a15425f90aa6a9"}
Oct 31 05:05:23.325564 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.325142    6931 scope.go:122] "RemoveContainer" containerID="22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"
Oct 31 05:05:23.336303 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.336282    6931 scope.go:122] "RemoveContainer" containerID="22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"
Oct 31 05:05:23.336584 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:23.336566    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058\": container with ID starting with 22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058 not found: ID does not exist" containerID="22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"
Oct 31 05:05:23.336627 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.336593    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058"} err="failed to get container status \"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058\": rpc error: code = NotFound desc = could not find container \"22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058\": container with ID starting with 22f3d596a62bdd7b8833c44f1fe2a1f9bca7df7d25d664763de03100e756b058 not found: ID does not exist"
Oct 31 05:05:23.377142 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.377106    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 05:05:23.389055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.389026    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-m56vf"]
Oct 31 05:05:23.585230 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.575802    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:23.869983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.869906    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 05:05:23.870145 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:23.870092    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy" containerID="cri-o://eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6" gracePeriod=30
Oct 31 05:05:24.127268 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:24.127185    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" path="/var/lib/kubelet/pods/e717b2a3-78b9-451e-b35a-26987837c05a/volumes"
Oct 31 05:05:24.328368 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:24.328333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:26.206651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:26.206615    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:27.524195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:27.523130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:28.386857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:28.386816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:29.141263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.141236    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 05:05:29.273056 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.272956    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273055    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273106    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273152    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273224    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273530 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273291    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:29.273530 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273303    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273530 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273452    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273713 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273565    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:29.273713 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273611    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273713 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273666    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\") pod \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\" (UID: \"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2\") "
Oct 31 05:05:29.273882 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273775    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:29.274019 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.273985    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:29.274361 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274314    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:29.274921 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274879    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.274921 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274908    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.274921 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274922    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.275175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274936    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.275175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.274952    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.275864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.275805    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:29.275864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.275831    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:05:29.276022 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.275868    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "kube-api-access-5l56j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:29.276022 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.275904    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy" pod "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" (UID: "f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:29.376437 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.376401    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l56j\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-kube-api-access-5l56j\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.376437 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.376437    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.376657 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.376449    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.376657 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.376461    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:29.378726 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.378698    6931 generic.go:356] "Generic (PLEG): container finished" podID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerID="eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6" exitCode=0
Oct 31 05:05:29.378857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.378756    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"
Oct 31 05:05:29.378857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.378797    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" event={"ID":"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2","Type":"ContainerDied","Data":"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"}
Oct 31 05:05:29.378947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.378856    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v" event={"ID":"f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2","Type":"ContainerDied","Data":"16f72b1bf253cf4ead32414e44ba1cdc780f48394e21f3752b7111e2114acfa3"}
Oct 31 05:05:29.378947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.378880    6931 scope.go:122] "RemoveContainer" containerID="eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"
Oct 31 05:05:29.390009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.389987    6931 scope.go:122] "RemoveContainer" containerID="eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"
Oct 31 05:05:29.390379 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:29.390360    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6\": container with ID starting with eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6 not found: ID does not exist" containerID="eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"
Oct 31 05:05:29.390441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.390389    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6"} err="failed to get container status \"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6\": rpc error: code = NotFound desc = could not find container \"eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6\": container with ID starting with eedd87b2bf30ffd2c61d118919ac65834443585acaef9fda989f423c53899ed6 not found: ID does not exist"
Oct 31 05:05:29.431081 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.431047    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 05:05:29.434951 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:29.434926    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-kdw2v"]
Oct 31 05:05:30.123960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:30.123917    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" path="/var/lib/kubelet/pods/f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2/volumes"
Oct 31 05:05:30.312263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:30.312227    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:32.168703 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:32.168662    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:34.017779 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:34.017742    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:35.875745 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:35.875710    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:37.036818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:37.036780    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:37.533105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:37.533049    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:37.925018 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:37.924940    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:39.341976 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.341932    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 05:05:39.342573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.342240    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" podUID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" containerName="maas-api" containerID="cri-o://3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd" gracePeriod=30
Oct 31 05:05:39.512766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.512724    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:39.652884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.650073    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 05:05:39.728462 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.728421    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\") pod \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") "
Oct 31 05:05:39.728676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.728522    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") pod \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") "
Oct 31 05:05:39.728752 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.728707    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\" (UniqueName: \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\") pod \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") "
Oct 31 05:05:39.728814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.728753    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\") pod \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\" (UID: \"0eb0ef43-32b6-4858-9c03-b309b6fa1d82\") "
Oct 31 05:05:39.729221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.729197    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca" pod "0eb0ef43-32b6-4858-9c03-b309b6fa1d82" (UID: "0eb0ef43-32b6-4858-9c03-b309b6fa1d82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:39.731045 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.731024    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8" pod "0eb0ef43-32b6-4858-9c03-b309b6fa1d82" (UID: "0eb0ef43-32b6-4858-9c03-b309b6fa1d82"). InnerVolumeSpecName "kube-api-access-lm4k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:39.738754 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.738662    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs" pod "0eb0ef43-32b6-4858-9c03-b309b6fa1d82" (UID: "0eb0ef43-32b6-4858-9c03-b309b6fa1d82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:39.742069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.740642    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls" pod "0eb0ef43-32b6-4858-9c03-b309b6fa1d82" (UID: "0eb0ef43-32b6-4858-9c03-b309b6fa1d82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:39.836199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.836131    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4k8\" (UniqueName: \"kubernetes.io/projected/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-kube-api-access-lm4k8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:39.836199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.836192    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:39.836561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.836210    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:39.836561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:39.836226    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0ef43-32b6-4858-9c03-b309b6fa1d82-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:40.395731 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.395643    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:40.457117 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.457084    6931 generic.go:356] "Generic (PLEG): container finished" podID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" containerID="3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd" exitCode=0
Oct 31 05:05:40.457315 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.457130    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"
Oct 31 05:05:40.457315 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.457199    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" event={"ID":"0eb0ef43-32b6-4858-9c03-b309b6fa1d82","Type":"ContainerDied","Data":"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"}
Oct 31 05:05:40.457315 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.457245    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc" event={"ID":"0eb0ef43-32b6-4858-9c03-b309b6fa1d82","Type":"ContainerDied","Data":"35491fd359259f15027b6e11fd90def09146e07818ecfdba0081985b322955e7"}
Oct 31 05:05:40.457315 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.457271    6931 scope.go:122] "RemoveContainer" containerID="3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"
Oct 31 05:05:40.469938 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.469880    6931 scope.go:122] "RemoveContainer" containerID="3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"
Oct 31 05:05:40.470456 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:40.470436    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd\": container with ID starting with 3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd not found: ID does not exist" containerID="3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"
Oct 31 05:05:40.470557 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.470465    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd"} err="failed to get container status \"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd\": rpc error: code = NotFound desc = could not find container \"3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd\": container with ID starting with 3a47b1b43257c05ef4ab2ae05faf243e708973438ddad94828cc44c53390eddd not found: ID does not exist"
Oct 31 05:05:40.508095 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.508055    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 05:05:40.519596 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:40.519564    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-171edd-b5875b446-scqgc"]
Oct 31 05:05:40.760417 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:40.760335    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:05:42.123286 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:42.123248    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" path="/var/lib/kubelet/pods/0eb0ef43-32b6-4858-9c03-b309b6fa1d82/volumes"
Oct 31 05:05:42.213121 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:42.213085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:43.238428 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:43.238391    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:43.443466 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:43.443422    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:44.121760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:44.121727    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:44.349305 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:44.349268    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:44.571365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:44.571323    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:44.792642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:44.792602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:44.998330 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:44.998244    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:45.559470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:45.559424    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:45.969578 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:45.969497    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:46.183705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:46.183669    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:46.432814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:46.432778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:47.939015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:47.938974    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:49.699926 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:49.699885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:50.610731 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:50.610678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:51.571209 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:51.571172    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:51.583352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:51.583323    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 05:05:51.612968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:51.612934    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 05:05:51.840676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:51.840576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:52.050873 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.050834    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:52.258658 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.258572    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:52.403107 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.403074    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 05:05:52.463541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.463509    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 05:05:52.680382 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.680342    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:52.890869 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:52.890830    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:53.123715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.123681    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:53.440104 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.439856    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:53.653627 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.653593    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:53.861453 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.861417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:53.977870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.977835    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 05:05:53.978072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:53.978052    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerName="discovery" containerID="cri-o://69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24" gracePeriod=30
Oct 31 05:05:54.252737 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.252710    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 05:05:54.436813 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.436720    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.436813 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.436788    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437066 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.436832    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437066 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.436860    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437066 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.436902    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.437147    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.437204    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\") pod \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\" (UID: \"4bb7d9e7-1ecc-4b76-97d3-770935fe5039\") "
Oct 31 05:05:54.437415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.437297    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:05:54.437781 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.437756    6931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-ca-configmap\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.439531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.439493    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:54.439717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.439694    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:54.439814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.439737    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:54.439814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.439755    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:05:54.439972 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.439954    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:05:54.440055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.440032    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm" pod "4bb7d9e7-1ecc-4b76-97d3-770935fe5039" (UID: "4bb7d9e7-1ecc-4b76-97d3-770935fe5039"). InnerVolumeSpecName "kube-api-access-bnjmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:05:54.539649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539615    6931 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-csr-dns-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.539649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539643    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjmm\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-kube-api-access-bnjmm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.539649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539654    6931 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-kubeconfig\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.539897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539664    6931 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-cacerts\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.539897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539674    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.539897 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.539685    6931 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb7d9e7-1ecc-4b76-97d3-770935fe5039-local-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:05:54.559823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.559788    6931 generic.go:356] "Generic (PLEG): container finished" podID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerID="69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24" exitCode=0
Oct 31 05:05:54.560031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.559842    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"
Oct 31 05:05:54.560031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.559875    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" event={"ID":"4bb7d9e7-1ecc-4b76-97d3-770935fe5039","Type":"ContainerDied","Data":"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"}
Oct 31 05:05:54.560031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.559933    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm" event={"ID":"4bb7d9e7-1ecc-4b76-97d3-770935fe5039","Type":"ContainerDied","Data":"29dbbcfa937af098a9e02d4fdc395ec5d5bec3f63e58529f6d113117c032e642"}
Oct 31 05:05:54.560031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.559955    6931 scope.go:122] "RemoveContainer" containerID="69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"
Oct 31 05:05:54.575144 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.575122    6931 scope.go:122] "RemoveContainer" containerID="69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"
Oct 31 05:05:54.575432 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:05:54.575412    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24\": container with ID starting with 69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24 not found: ID does not exist" containerID="69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"
Oct 31 05:05:54.575497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.575442    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24"} err="failed to get container status \"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24\": rpc error: code = NotFound desc = could not find container \"69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24\": container with ID starting with 69e90bb6da8f5da36ba086e8430177f7fee379738e972b16665706b07a69fb24 not found: ID does not exist"
Oct 31 05:05:54.612136 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.612103    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 05:05:54.626080 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:54.626050    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7wbkm"]
Oct 31 05:05:55.750440 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:55.750404    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:56.124000 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:56.123967    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" path="/var/lib/kubelet/pods/4bb7d9e7-1ecc-4b76-97d3-770935fe5039/volumes"
Oct 31 05:05:56.584036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:56.583984    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing" containerID="cri-o://463e3fcdd6c14e511402f9642fb0c933aef9c7f5116bfe3c5521298391460013" gracePeriod=25
Oct 31 05:05:56.613631 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:56.613573    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing" containerID="cri-o://87cbcfd7c2734e59cfc33fdaa0eae8561c3789be63c19684bc41f8de9efe9d15" gracePeriod=25
Oct 31 05:05:57.403675 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:57.403613    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing" containerID="cri-o://63bfe7bd046ab1e439d91c1bf18bdb412458a8ea0e2554aaca9d3f84a35e8043" gracePeriod=25
Oct 31 05:05:57.464786 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:57.464739    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing" containerID="cri-o://cbb01d5f11964e197711c63d33fca6b2176a635c86fc5f397d64a1c5378b7833" gracePeriod=25
Oct 31 05:05:57.536694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:57.536655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:58.213042 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:58.213007    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:59.262806 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:59.262764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:59.476861 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:59.476811    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:05:59.694413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:05:59.694372    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:00.646518 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:00.646478    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 05:06:00.647014 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:00.646700    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" podUID="c02c0998-b492-4575-ba14-1bf8076fa354" containerName="maas-api" containerID="cri-o://a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730" gracePeriod=30
Oct 31 05:06:00.823736 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:00.823695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:00.855733 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:00.855694    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:00.855887 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:00.855702    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:01.590861 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:01.590817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:01.682802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:01.682756    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:01.683208 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:01.682775    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:01.917752 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:01.917724    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 05:06:02.036895 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.036861    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\") pod \"c02c0998-b492-4575-ba14-1bf8076fa354\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") "
Oct 31 05:06:02.037105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.036975    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\") pod \"c02c0998-b492-4575-ba14-1bf8076fa354\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") "
Oct 31 05:06:02.037105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.037022    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\") pod \"c02c0998-b492-4575-ba14-1bf8076fa354\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") "
Oct 31 05:06:02.037264 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.037217    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\" (UniqueName: \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\") pod \"c02c0998-b492-4575-ba14-1bf8076fa354\" (UID: \"c02c0998-b492-4575-ba14-1bf8076fa354\") "
Oct 31 05:06:02.037327 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.037301    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca" pod "c02c0998-b492-4575-ba14-1bf8076fa354" (UID: "c02c0998-b492-4575-ba14-1bf8076fa354"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:02.037621 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.037600    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c02c0998-b492-4575-ba14-1bf8076fa354-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:02.039207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.039166    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls" pod "c02c0998-b492-4575-ba14-1bf8076fa354" (UID: "c02c0998-b492-4575-ba14-1bf8076fa354"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:02.039361 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.039338    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s" pod "c02c0998-b492-4575-ba14-1bf8076fa354" (UID: "c02c0998-b492-4575-ba14-1bf8076fa354"). InnerVolumeSpecName "kube-api-access-ztw4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:02.039440 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.039422    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs" pod "c02c0998-b492-4575-ba14-1bf8076fa354" (UID: "c02c0998-b492-4575-ba14-1bf8076fa354"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:02.138649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.138581    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:02.138649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.138610    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02c0998-b492-4575-ba14-1bf8076fa354-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:02.138649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.138623    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztw4s\" (UniqueName: \"kubernetes.io/projected/c02c0998-b492-4575-ba14-1bf8076fa354-kube-api-access-ztw4s\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:02.630473 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.630439    6931 generic.go:356] "Generic (PLEG): container finished" podID="c02c0998-b492-4575-ba14-1bf8076fa354" containerID="a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730" exitCode=0
Oct 31 05:06:02.630473 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.630484    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" event={"ID":"c02c0998-b492-4575-ba14-1bf8076fa354","Type":"ContainerDied","Data":"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"}
Oct 31 05:06:02.630712 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.630521    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h" event={"ID":"c02c0998-b492-4575-ba14-1bf8076fa354","Type":"ContainerDied","Data":"b88109387ae343d4ce561fd37b14bdf14698017d2a67d4e7a4c8e29fc51ecb3d"}
Oct 31 05:06:02.630712 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.630542    6931 scope.go:122] "RemoveContainer" containerID="a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"
Oct 31 05:06:02.630712 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.630485    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"
Oct 31 05:06:02.644195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.644171    6931 scope.go:122] "RemoveContainer" containerID="a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"
Oct 31 05:06:02.644556 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:02.644536    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730\": container with ID starting with a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730 not found: ID does not exist" containerID="a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"
Oct 31 05:06:02.644608 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.644565    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730"} err="failed to get container status \"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730\": rpc error: code = NotFound desc = could not find container \"a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730\": container with ID starting with a03e1f18c5d17aa0479a1afdfd361610cf0249c6f56397c14d8782056ba0a730 not found: ID does not exist"
Oct 31 05:06:02.690516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.690471    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 05:06:02.708905 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.708864    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"]
Oct 31 05:06:02.710105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710076    6931 cpu_manager.go:414] "containerMap: removing container" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy"
Oct 31 05:06:02.710284 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710269    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy"
Oct 31 05:06:02.710450 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710435    6931 cpu_manager.go:414] "containerMap: removing container" podUID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" containerName="maas-api"
Oct 31 05:06:02.710555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710543    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" containerName="maas-api"
Oct 31 05:06:02.710706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710692    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" containerName="maas-api"
Oct 31 05:06:02.710801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710790    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" containerName="maas-api"
Oct 31 05:06:02.710892 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710882    6931 cpu_manager.go:414] "containerMap: removing container" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing"
Oct 31 05:06:02.710968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.710958    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing"
Oct 31 05:06:02.711063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711053    6931 cpu_manager.go:414] "containerMap: removing container" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy"
Oct 31 05:06:02.711130 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711121    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy"
Oct 31 05:06:02.711234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711223    6931 cpu_manager.go:414] "containerMap: removing container" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing"
Oct 31 05:06:02.711323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711312    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing"
Oct 31 05:06:02.711404 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711395    6931 cpu_manager.go:414] "containerMap: removing container" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerName="discovery"
Oct 31 05:06:02.711476 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711464    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerName="discovery"
Oct 31 05:06:02.711547 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711539    6931 cpu_manager.go:414] "containerMap: removing container" podUID="24637e83-a5d7-4480-b83d-1c9327565daa" containerName="main"
Oct 31 05:06:02.711617 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711608    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24637e83-a5d7-4480-b83d-1c9327565daa" containerName="main"
Oct 31 05:06:02.711699 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711690    6931 cpu_manager.go:414] "containerMap: removing container" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing"
Oct 31 05:06:02.711768 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711759    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing"
Oct 31 05:06:02.711846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711837    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" containerName="maas-api"
Oct 31 05:06:02.711920 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711911    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" containerName="maas-api"
Oct 31 05:06:02.711997 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.711988    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing"
Oct 31 05:06:02.712065 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712055    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing"
Oct 31 05:06:02.712147 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712138    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing"
Oct 31 05:06:02.712240 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712228    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing"
Oct 31 05:06:02.712344 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712335    6931 cpu_manager.go:414] "containerMap: removing container" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy"
Oct 31 05:06:02.712417 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712407    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy"
Oct 31 05:06:02.712492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712484    6931 cpu_manager.go:414] "containerMap: removing container" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy"
Oct 31 05:06:02.712569 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712560    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy"
Oct 31 05:06:02.712661 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712652    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c02c0998-b492-4575-ba14-1bf8076fa354" containerName="maas-api"
Oct 31 05:06:02.712729 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.712721    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02c0998-b492-4575-ba14-1bf8076fa354" containerName="maas-api"
Oct 31 05:06:02.713039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713025    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="e717b2a3-78b9-451e-b35a-26987837c05a" containerName="istio-proxy"
Oct 31 05:06:02.713136 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713127    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c539ae-d39a-42c8-b1e3-2754023442fc" containerName="payload-processing"
Oct 31 05:06:02.713239 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713229    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb7d9e7-1ecc-4b76-97d3-770935fe5039" containerName="discovery"
Oct 31 05:06:02.713314 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713306    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="24637e83-a5d7-4480-b83d-1c9327565daa" containerName="main"
Oct 31 05:06:02.713385 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713376    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f11c6a-23af-4678-932c-1aa7c4b85242" containerName="payload-pre-processing"
Oct 31 05:06:02.713455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713446    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278703c-e2f4-48f7-b7d0-7842f5f65a29" containerName="maas-api"
Oct 31 05:06:02.713542 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713532    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3d97e2-092e-4a1b-8509-94d92225e1b6" containerName="payload-processing"
Oct 31 05:06:02.713613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713605    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc6db11-c632-4c17-93b1-ac9c8e2cc246" containerName="payload-processing"
Oct 31 05:06:02.713689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713680    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c9d633-4cb4-4997-b95a-ecc9c0acbcc2" containerName="istio-proxy"
Oct 31 05:06:02.713763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713755    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ad350c-6196-420e-9d8b-2210b5ff7152" containerName="istio-proxy"
Oct 31 05:06:02.713840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713832    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02c0998-b492-4575-ba14-1bf8076fa354" containerName="maas-api"
Oct 31 05:06:02.713914 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.713906    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8d7d49-d719-4fed-971b-89f538d0e39c" containerName="istio-proxy"
Oct 31 05:06:02.714053 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.714041    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda4ea64-5dd6-421e-bfc5-e3b290990ca6" containerName="maas-api"
Oct 31 05:06:02.714194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.714181    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb0ef43-32b6-4858-9c03-b309b6fa1d82" containerName="maas-api"
Oct 31 05:06:02.714309 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.714297    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e60e01-bce0-4ea9-aad3-f888dda53189" containerName="payload-pre-processing"
Oct 31 05:06:02.725370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.725338    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04d111-6d7544bddf-4g64h"]
Oct 31 05:06:02.726765 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.725683    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.745876 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.745846    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"]
Oct 31 05:06:02.846985 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.846940    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.846992    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6pc\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-kube-api-access-cj6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847087    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847310 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847216    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847310 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847290    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847425 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847310    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847425 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847401    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847439    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.847516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.847493    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949130    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949204    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6pc\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-kube-api-access-cj6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949225    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949286    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949332    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949350    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949395    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949430    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949459    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949830 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949747    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949830 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949780    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949922 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949902    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.949964 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.949943    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.950216 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.950194    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.951628 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.951606    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.951868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.951848    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.972564 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.972534    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6pc\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-kube-api-access-cj6pc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:02.972717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:02.972688    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7b9e7bd-eab8-43ed-a754-ed016aeab9c7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5znqb\" (UID: \"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:03.049984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.049945    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:03.131335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.131296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:03.370501 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.370476    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"]
Oct 31 05:06:03.373479 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:06:03.373451    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b9e7bd_eab8_43ed_a754_ed016aeab9c7.slice/crio-46f0f28a10adad80ad0cf62364b2af02e09d518af2b2ab3ff1bb0518bb49a150 WatchSource:0}: Error finding container 46f0f28a10adad80ad0cf62364b2af02e09d518af2b2ab3ff1bb0518bb49a150: Status 404 returned error can't find the container with id 46f0f28a10adad80ad0cf62364b2af02e09d518af2b2ab3ff1bb0518bb49a150
Oct 31 05:06:03.376218 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.376147    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:03.376295 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.376280    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:03.376339 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.376316    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:03.458027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.457992    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:03.642059 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.641969    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" event={"ID":"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7","Type":"ContainerStarted","Data":"f3005842fd1c1c4d8abeadb3e3d7a6692150e2dd1328c1b8ad552ca2d86b3f95"}
Oct 31 05:06:03.642059 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.642018    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" event={"ID":"a7b9e7bd-eab8-43ed-a754-ed016aeab9c7","Type":"ContainerStarted","Data":"46f0f28a10adad80ad0cf62364b2af02e09d518af2b2ab3ff1bb0518bb49a150"}
Oct 31 05:06:03.687351 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:03.687287    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podStartSLOduration=1.687268752 podStartE2EDuration="1.687268752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:03.685993747 +0000 UTC m=+3659.718993447" watchObservedRunningTime="2026-10-31 05:06:03.687268752 +0000 UTC m=+3659.720268439"
Oct 31 05:06:04.052213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:04.052169    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:04.052734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:04.052309    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:04.052734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:04.052356    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:04.153211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:04.152355    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02c0998-b492-4575-ba14-1bf8076fa354" path="/var/lib/kubelet/pods/c02c0998-b492-4575-ba14-1bf8076fa354/volumes"
Oct 31 05:06:05.051600 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:05.051549    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:05.051825 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:05.051620    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:05.509184 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:05.509138    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:05.801492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:05.801450    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:06.013039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:06.013005    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:06.051736 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:06.051634    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:06.051736 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:06.051711    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:06.652808 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:06.652772    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:07.004109 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:07.004029    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:07.056234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:07.054372    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:07.056234 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:07.054434    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:07.795172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:07.795120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:08.024001 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:08.023939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:08.051241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:08.050573    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:08.051241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:08.050648    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:09.051573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:09.051533    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:09.051980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:09.051604    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:09.827323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:09.827287    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:10.050546 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:10.050502    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 05:06:10.050734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:10.050573    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 05:06:10.050734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:10.050610    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:10.761306 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:10.761262    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:06:10.854575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:10.854531    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:10.854785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:10.854724    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:11.683430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.683389    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:11.683620 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.683400    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:06:11.895651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.895614    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:11.896417 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.896358    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:11.897194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.897149    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb"
Oct 31 05:06:11.899644 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:11.899619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:13.951710 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:13.951670    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:14.988050 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:14.988004    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:15.192996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:15.192953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:16.103437 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.103395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:16.739715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.739673    6931 generic.go:356] "Generic (PLEG): container finished" podID="7597c178-3922-4646-9ee2-75b5474fec53" containerID="463e3fcdd6c14e511402f9642fb0c933aef9c7f5116bfe3c5521298391460013" exitCode=0
Oct 31 05:06:16.739878 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.739746    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" event={"ID":"7597c178-3922-4646-9ee2-75b5474fec53","Type":"ContainerDied","Data":"463e3fcdd6c14e511402f9642fb0c933aef9c7f5116bfe3c5521298391460013"}
Oct 31 05:06:16.741777 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.741751    6931 generic.go:356] "Generic (PLEG): container finished" podID="200cdbab-5ef1-406c-9397-b89894111a40" containerID="87cbcfd7c2734e59cfc33fdaa0eae8561c3789be63c19684bc41f8de9efe9d15" exitCode=0
Oct 31 05:06:16.741875 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.741799    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" event={"ID":"200cdbab-5ef1-406c-9397-b89894111a40","Type":"ContainerDied","Data":"87cbcfd7c2734e59cfc33fdaa0eae8561c3789be63c19684bc41f8de9efe9d15"}
Oct 31 05:06:16.904796 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.904766    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 05:06:16.928643 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:16.928618    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 05:06:17.057135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057102    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\") pod \"7597c178-3922-4646-9ee2-75b5474fec53\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") "
Oct 31 05:06:17.057367 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057215    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\" (UniqueName: \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\") pod \"7597c178-3922-4646-9ee2-75b5474fec53\" (UID: \"7597c178-3922-4646-9ee2-75b5474fec53\") "
Oct 31 05:06:17.057367 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057269    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\") pod \"200cdbab-5ef1-406c-9397-b89894111a40\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") "
Oct 31 05:06:17.057495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057457    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\" (UniqueName: \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\") pod \"200cdbab-5ef1-406c-9397-b89894111a40\" (UID: \"200cdbab-5ef1-406c-9397-b89894111a40\") "
Oct 31 05:06:17.057674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057643    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume" pod "7597c178-3922-4646-9ee2-75b5474fec53" (UID: "7597c178-3922-4646-9ee2-75b5474fec53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:17.057792 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057721    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume" pod "200cdbab-5ef1-406c-9397-b89894111a40" (UID: "200cdbab-5ef1-406c-9397-b89894111a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:17.057905 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057887    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7597c178-3922-4646-9ee2-75b5474fec53-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.057971 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.057908    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200cdbab-5ef1-406c-9397-b89894111a40-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.059423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.059400    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c" pod "7597c178-3922-4646-9ee2-75b5474fec53" (UID: "7597c178-3922-4646-9ee2-75b5474fec53"). InnerVolumeSpecName "kube-api-access-pzn8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:17.059508 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.059470    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c" pod "200cdbab-5ef1-406c-9397-b89894111a40" (UID: "200cdbab-5ef1-406c-9397-b89894111a40"). InnerVolumeSpecName "kube-api-access-mh88c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:17.159370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.159284    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzn8c\" (UniqueName: \"kubernetes.io/projected/7597c178-3922-4646-9ee2-75b5474fec53-kube-api-access-pzn8c\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.159370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.159313    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh88c\" (UniqueName: \"kubernetes.io/projected/200cdbab-5ef1-406c-9397-b89894111a40-kube-api-access-mh88c\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.749855 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.749819    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"]
Oct 31 05:06:17.751201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751173    6931 cpu_manager.go:414] "containerMap: removing container" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing"
Oct 31 05:06:17.751201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751200    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing"
Oct 31 05:06:17.751475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751245    6931 cpu_manager.go:414] "containerMap: removing container" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing"
Oct 31 05:06:17.751475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751257    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing"
Oct 31 05:06:17.751599 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751472    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="7597c178-3922-4646-9ee2-75b5474fec53" containerName="payload-processing"
Oct 31 05:06:17.751599 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.751496    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="200cdbab-5ef1-406c-9397-b89894111a40" containerName="payload-pre-processing"
Oct 31 05:06:17.754223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.753345    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"
Oct 31 05:06:17.755122 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.755100    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"
Oct 31 05:06:17.755798 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.755769    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh" event={"ID":"7597c178-3922-4646-9ee2-75b5474fec53","Type":"ContainerDied","Data":"70ebf5358133d13a2af123b5728c9faa0eeb6d26704bcc08358735cf60924344"}
Oct 31 05:06:17.755906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.755816    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj" event={"ID":"200cdbab-5ef1-406c-9397-b89894111a40","Type":"ContainerDied","Data":"40b844b6d02af8d989e355ad02ec80ea9d37ad37465246e9f80ea6ace3fd3af3"}
Oct 31 05:06:17.755906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.755849    6931 scope.go:122] "RemoveContainer" containerID="463e3fcdd6c14e511402f9642fb0c933aef9c7f5116bfe3c5521298391460013"
Oct 31 05:06:17.756032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.755906    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.756933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.756778    6931 generic.go:356] "Generic (PLEG): container finished" podID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerID="63bfe7bd046ab1e439d91c1bf18bdb412458a8ea0e2554aaca9d3f84a35e8043" exitCode=0
Oct 31 05:06:17.756933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.756860    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" event={"ID":"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a","Type":"ContainerDied","Data":"63bfe7bd046ab1e439d91c1bf18bdb412458a8ea0e2554aaca9d3f84a35e8043"}
Oct 31 05:06:17.756933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.756898    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw" event={"ID":"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a","Type":"ContainerDied","Data":"2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09"}
Oct 31 05:06:17.756933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.756922    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df55d631b58f8f0e715be4ad3a16c2d6694fb27b56045fde2092d23c83b8b09"
Oct 31 05:06:17.762114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.762088    6931 generic.go:356] "Generic (PLEG): container finished" podID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerID="cbb01d5f11964e197711c63d33fca6b2176a635c86fc5f397d64a1c5378b7833" exitCode=0
Oct 31 05:06:17.762276 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.762133    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" event={"ID":"98e01560-0500-47bd-8070-4c7e2a50b76c","Type":"ContainerDied","Data":"cbb01d5f11964e197711c63d33fca6b2176a635c86fc5f397d64a1c5378b7833"}
Oct 31 05:06:17.775577 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.775549    6931 scope.go:122] "RemoveContainer" containerID="87cbcfd7c2734e59cfc33fdaa0eae8561c3789be63c19684bc41f8de9efe9d15"
Oct 31 05:06:17.776832 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.776804    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"]
Oct 31 05:06:17.779486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.779442    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 05:06:17.824474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.824449    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 05:06:17.855173 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.855134    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 05:06:17.859095 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.859070    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1daf30-654cd76bb7-29rqh"]
Oct 31 05:06:17.869691 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.869663    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\") pod \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") "
Oct 31 05:06:17.869832 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.869818    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\" (UniqueName: \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\") pod \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\" (UID: \"bf7f1f45-03a8-4c35-9c8d-c3c9c287742a\") "
Oct 31 05:06:17.870037 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870014    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume" pod "bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" (UID: "bf7f1f45-03a8-4c35-9c8d-c3c9c287742a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:17.870087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870043    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870133 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870098    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870256 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870230    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870336 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870317    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870400    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-kube-api-access-hz8gh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870459 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870423    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870459 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870446    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870619 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870602    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870641    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.870795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.870780    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.871966 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.871945    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb" pod "bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" (UID: "bf7f1f45-03a8-4c35-9c8d-c3c9c287742a"). InnerVolumeSpecName "kube-api-access-9c5fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:17.891716 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.891687    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 05:06:17.898673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.898635    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1daf30-f769b9875-5xzlj"]
Oct 31 05:06:17.971340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971243    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\" (UniqueName: \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\") pod \"98e01560-0500-47bd-8070-4c7e2a50b76c\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") "
Oct 31 05:06:17.971570 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971514    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\") pod \"98e01560-0500-47bd-8070-4c7e2a50b76c\" (UID: \"98e01560-0500-47bd-8070-4c7e2a50b76c\") "
Oct 31 05:06:17.971769 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971742    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971835    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-kube-api-access-hz8gh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971866    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971901    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.971907    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume" pod "98e01560-0500-47bd-8070-4c7e2a50b76c" (UID: "98e01560-0500-47bd-8070-4c7e2a50b76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972030    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972070    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972189    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972238    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972273    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972368    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e01560-0500-47bd-8070-4c7e2a50b76c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.972587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.972412    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5fb\" (UniqueName: \"kubernetes.io/projected/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a-kube-api-access-9c5fb\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:17.974196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.973828    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.974488 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.974454    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.974654 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.974629    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.974974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.974854    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.975915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.975709    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.976111 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.976088    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2" pod "98e01560-0500-47bd-8070-4c7e2a50b76c" (UID: "98e01560-0500-47bd-8070-4c7e2a50b76c"). InnerVolumeSpecName "kube-api-access-2mtb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:17.977083 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.977037    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.977605 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.977565    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.992561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.992522    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:17.996061 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.996036    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:17.996398 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:17.996371    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/de27e8de-c83f-48d2-b90f-c1bad562a1d0-kube-api-access-hz8gh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-jd8wd\" (UID: \"de27e8de-c83f-48d2-b90f-c1bad562a1d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:18.073952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.073912    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtb2\" (UniqueName: \"kubernetes.io/projected/98e01560-0500-47bd-8070-4c7e2a50b76c-kube-api-access-2mtb2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:18.123762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.123719    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:18.131631 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.131603    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200cdbab-5ef1-406c-9397-b89894111a40" path="/var/lib/kubelet/pods/200cdbab-5ef1-406c-9397-b89894111a40/volumes"
Oct 31 05:06:18.132027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.132014    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7597c178-3922-4646-9ee2-75b5474fec53" path="/var/lib/kubelet/pods/7597c178-3922-4646-9ee2-75b5474fec53/volumes"
Oct 31 05:06:18.452626 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.452600    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"]
Oct 31 05:06:18.455461 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:06:18.455435    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde27e8de_c83f_48d2_b90f_c1bad562a1d0.slice/crio-c474218d54b3a9ae5303baf78ddcd7131d90d1699110ac239188d36efbf7d203 WatchSource:0}: Error finding container c474218d54b3a9ae5303baf78ddcd7131d90d1699110ac239188d36efbf7d203: Status 404 returned error can't find the container with id c474218d54b3a9ae5303baf78ddcd7131d90d1699110ac239188d36efbf7d203
Oct 31 05:06:18.458358 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.458317    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:18.458489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.458393    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:18.458489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.458424    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:06:18.771333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.771230    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" event={"ID":"de27e8de-c83f-48d2-b90f-c1bad562a1d0","Type":"ContainerStarted","Data":"921db2ed4bb273883312626f0de94f8a3b443426b2ce4c09e264c51669094f33"}
Oct 31 05:06:18.771333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.771291    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" event={"ID":"de27e8de-c83f-48d2-b90f-c1bad562a1d0","Type":"ContainerStarted","Data":"c474218d54b3a9ae5303baf78ddcd7131d90d1699110ac239188d36efbf7d203"}
Oct 31 05:06:18.772598 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.772580    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"
Oct 31 05:06:18.772748 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.772622    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf" event={"ID":"98e01560-0500-47bd-8070-4c7e2a50b76c","Type":"ContainerDied","Data":"6e5b2c984e28e9432c4e3aefacc6250cb038b58e5236a43ae6e3af7ab901ae46"}
Oct 31 05:06:18.772748 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.772685    6931 scope.go:122] "RemoveContainer" containerID="cbb01d5f11964e197711c63d33fca6b2176a635c86fc5f397d64a1c5378b7833"
Oct 31 05:06:18.773961 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.773708    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"
Oct 31 05:06:18.820010 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.819937    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podStartSLOduration=1.819919566 podStartE2EDuration="1.819919566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:06:18.815669809 +0000 UTC m=+3674.848669510" watchObservedRunningTime="2026-10-31 05:06:18.819919566 +0000 UTC m=+3674.852919250"
Oct 31 05:06:18.857142 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.857109    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 05:06:18.862276 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.862245    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb346b-7f79f66479-vn5cf"]
Oct 31 05:06:18.894481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.894439    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 05:06:18.898435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:18.898397    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fb346b-5fbbccc457-skfrw"]
Oct 31 05:06:19.127871 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:19.127834    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:19.129819 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:19.129791    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:19.129951 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:19.129862    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:19.904299 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:19.904253    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:20.123096 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:20.123061    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" path="/var/lib/kubelet/pods/98e01560-0500-47bd-8070-4c7e2a50b76c/volumes"
Oct 31 05:06:20.123503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:20.123488    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" path="/var/lib/kubelet/pods/bf7f1f45-03a8-4c35-9c8d-c3c9c287742a/volumes"
Oct 31 05:06:20.144182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:20.133599    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:20.144182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:20.133662    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:21.147182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:21.144601    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:21.147182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:21.144666    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:21.353011 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:21.352974    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:22.034827 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:22.034789    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:22.138189 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:22.130550    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:22.138189 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:22.130612    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:23.142571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:23.142530    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:23.143148 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:23.143115    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:24.105977 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:24.105933    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:24.139182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:24.137588    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:24.139182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:24.137660    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:25.128213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:25.128123    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:25.128634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:25.128241    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:26.006840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:26.006799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:26.127289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:26.127250    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 05:06:26.127460 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:26.127318    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 05:06:27.338721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:27.338672    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:27.340082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:27.340056    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:27.340611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:27.340591    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd"
Oct 31 05:06:27.829763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:27.829722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:29.231282 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.231242    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 05:06:29.231786 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.231541    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" podUID="4e483801-b8a1-44a1-8999-efd4a8faf25d" containerName="maas-api" containerID="cri-o://226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584" gracePeriod=30
Oct 31 05:06:29.402954 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.402913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:29.518429 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.518403    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 05:06:29.641245 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.641213    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\") pod \"4e483801-b8a1-44a1-8999-efd4a8faf25d\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") "
Oct 31 05:06:29.641435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.641344    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\") pod \"4e483801-b8a1-44a1-8999-efd4a8faf25d\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") "
Oct 31 05:06:29.641435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.641399    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") pod \"4e483801-b8a1-44a1-8999-efd4a8faf25d\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") "
Oct 31 05:06:29.641548 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.641531    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\" (UniqueName: \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\") pod \"4e483801-b8a1-44a1-8999-efd4a8faf25d\" (UID: \"4e483801-b8a1-44a1-8999-efd4a8faf25d\") "
Oct 31 05:06:29.641656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.641632    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca" pod "4e483801-b8a1-44a1-8999-efd4a8faf25d" (UID: "4e483801-b8a1-44a1-8999-efd4a8faf25d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:29.642217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.642194    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e483801-b8a1-44a1-8999-efd4a8faf25d-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:29.643583 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.643538    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw" pod "4e483801-b8a1-44a1-8999-efd4a8faf25d" (UID: "4e483801-b8a1-44a1-8999-efd4a8faf25d"). InnerVolumeSpecName "kube-api-access-rzljw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:29.643673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.643613    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs" pod "4e483801-b8a1-44a1-8999-efd4a8faf25d" (UID: "4e483801-b8a1-44a1-8999-efd4a8faf25d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:29.643673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.643628    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls" pod "4e483801-b8a1-44a1-8999-efd4a8faf25d" (UID: "4e483801-b8a1-44a1-8999-efd4a8faf25d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:29.743740 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.743656    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzljw\" (UniqueName: \"kubernetes.io/projected/4e483801-b8a1-44a1-8999-efd4a8faf25d-kube-api-access-rzljw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:29.743740 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.743687    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:29.743740 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.743696    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e483801-b8a1-44a1-8999-efd4a8faf25d-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:29.848251 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.848214    6931 generic.go:356] "Generic (PLEG): container finished" podID="4e483801-b8a1-44a1-8999-efd4a8faf25d" containerID="226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584" exitCode=0
Oct 31 05:06:29.848409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.848270    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"
Oct 31 05:06:29.848409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.848312    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" event={"ID":"4e483801-b8a1-44a1-8999-efd4a8faf25d","Type":"ContainerDied","Data":"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"}
Oct 31 05:06:29.848409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.848378    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2" event={"ID":"4e483801-b8a1-44a1-8999-efd4a8faf25d","Type":"ContainerDied","Data":"8f452bb9d5a218c8d2ff447706c7d6d99a7f62b3123ac1702e65dbffdebab604"}
Oct 31 05:06:29.848409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.848403    6931 scope.go:122] "RemoveContainer" containerID="226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"
Oct 31 05:06:29.859620 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.859598    6931 scope.go:122] "RemoveContainer" containerID="226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"
Oct 31 05:06:29.859906 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:29.859885    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584\": container with ID starting with 226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584 not found: ID does not exist" containerID="226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"
Oct 31 05:06:29.859979 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.859913    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584"} err="failed to get container status \"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584\": rpc error: code = NotFound desc = could not find container \"226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584\": container with ID starting with 226d0228bd41132b3f74394d670dcd8afa93d0f9767fa9d66c71cc16305f5584 not found: ID does not exist"
Oct 31 05:06:29.900190 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.900147    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 05:06:29.903663 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:29.903640    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1daf30-864d4f4889-gjdf2"]
Oct 31 05:06:30.123725 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:30.123682    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e483801-b8a1-44a1-8999-efd4a8faf25d" path="/var/lib/kubelet/pods/4e483801-b8a1-44a1-8999-efd4a8faf25d/volumes"
Oct 31 05:06:30.222143 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:30.222106    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:32.094594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:32.094550    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:32.330914 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:32.330872    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:32.895056 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:32.895014    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:33.243435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:33.243321    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:34.068880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:34.068846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:34.283122 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:34.283071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:36.631085 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:36.631044    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:38.425524 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:38.425477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:40.243464 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:40.243427    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:40.546924 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:40.546885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:40.760383 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:40.760340    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:06:40.985867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:40.985779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:41.202347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:41.202307    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:42.266734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:42.266698    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:43.884899 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:43.884866    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 05:06:43.885353 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:43.885120    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" podUID="529a5853-adf2-4250-96d1-9a9fc9d251e2" containerName="maas-api" containerID="cri-o://f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96" gracePeriod=30
Oct 31 05:06:44.052696 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.052647    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:44.186213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.183554    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 05:06:44.233217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.229238    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\") pod \"529a5853-adf2-4250-96d1-9a9fc9d251e2\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") "
Oct 31 05:06:44.233217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.229334    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\" (UniqueName: \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\") pod \"529a5853-adf2-4250-96d1-9a9fc9d251e2\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") "
Oct 31 05:06:44.233217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.229409    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") pod \"529a5853-adf2-4250-96d1-9a9fc9d251e2\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") "
Oct 31 05:06:44.233217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.229608    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\") pod \"529a5853-adf2-4250-96d1-9a9fc9d251e2\" (UID: \"529a5853-adf2-4250-96d1-9a9fc9d251e2\") "
Oct 31 05:06:44.233217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.230762    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca" pod "529a5853-adf2-4250-96d1-9a9fc9d251e2" (UID: "529a5853-adf2-4250-96d1-9a9fc9d251e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:44.244199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.233841    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs" pod "529a5853-adf2-4250-96d1-9a9fc9d251e2" (UID: "529a5853-adf2-4250-96d1-9a9fc9d251e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:44.244199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.238619    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv" pod "529a5853-adf2-4250-96d1-9a9fc9d251e2" (UID: "529a5853-adf2-4250-96d1-9a9fc9d251e2"). InnerVolumeSpecName "kube-api-access-wfqbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:44.244199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.241309    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls" pod "529a5853-adf2-4250-96d1-9a9fc9d251e2" (UID: "529a5853-adf2-4250-96d1-9a9fc9d251e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:44.331887 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.331853    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:44.331887 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.331882    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfqbv\" (UniqueName: \"kubernetes.io/projected/529a5853-adf2-4250-96d1-9a9fc9d251e2-kube-api-access-wfqbv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:44.331887 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.331894    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/529a5853-adf2-4250-96d1-9a9fc9d251e2-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:44.332136 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.331902    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/529a5853-adf2-4250-96d1-9a9fc9d251e2-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:44.960343 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.960301    6931 generic.go:356] "Generic (PLEG): container finished" podID="529a5853-adf2-4250-96d1-9a9fc9d251e2" containerID="f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96" exitCode=0
Oct 31 05:06:44.960864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.960356    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"
Oct 31 05:06:44.960864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.960479    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" event={"ID":"529a5853-adf2-4250-96d1-9a9fc9d251e2","Type":"ContainerDied","Data":"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"}
Oct 31 05:06:44.960864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.960596    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg" event={"ID":"529a5853-adf2-4250-96d1-9a9fc9d251e2","Type":"ContainerDied","Data":"235176bbc192aa7b4d72e1ff1d618b48f6b93c24649f4bba19bf99c262be1bb5"}
Oct 31 05:06:44.960864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.960631    6931 scope.go:122] "RemoveContainer" containerID="f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"
Oct 31 05:06:44.973404 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.973379    6931 scope.go:122] "RemoveContainer" containerID="f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"
Oct 31 05:06:44.973726 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:44.973702    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96\": container with ID starting with f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96 not found: ID does not exist" containerID="f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"
Oct 31 05:06:44.973799 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:44.973740    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96"} err="failed to get container status \"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96\": rpc error: code = NotFound desc = could not find container \"f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96\": container with ID starting with f4108359bff22b9f17841068237415e8109b0271e62f495a81e0811ff8089d96 not found: ID does not exist"
Oct 31 05:06:45.011705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:45.011671    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 05:06:45.020467 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:45.020431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:45.022880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:45.022854    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-108ad8-7bb45ff58-786jg"]
Oct 31 05:06:46.126600 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:46.126563    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529a5853-adf2-4250-96d1-9a9fc9d251e2" path="/var/lib/kubelet/pods/529a5853-adf2-4250-96d1-9a9fc9d251e2/volumes"
Oct 31 05:06:46.210349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:46.210296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:46.794992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:46.794953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:48.166201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.166134    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 05:06:48.166750 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.166415    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" podUID="0c0a447a-fa13-43a3-8d81-6191d115ff17" containerName="maas-api" containerID="cri-o://46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09" gracePeriod=30
Oct 31 05:06:48.330233 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.330187    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:48.492242 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.492216    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 05:06:48.589552 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.589508    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\") pod \"0c0a447a-fa13-43a3-8d81-6191d115ff17\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") "
Oct 31 05:06:48.589735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.589655    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\") pod \"0c0a447a-fa13-43a3-8d81-6191d115ff17\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") "
Oct 31 05:06:48.589735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.589706    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\" (UniqueName: \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\") pod \"0c0a447a-fa13-43a3-8d81-6191d115ff17\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") "
Oct 31 05:06:48.589810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.589779    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\") pod \"0c0a447a-fa13-43a3-8d81-6191d115ff17\" (UID: \"0c0a447a-fa13-43a3-8d81-6191d115ff17\") "
Oct 31 05:06:48.590088 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.590057    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca" pod "0c0a447a-fa13-43a3-8d81-6191d115ff17" (UID: "0c0a447a-fa13-43a3-8d81-6191d115ff17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:06:48.590374 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.590356    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a447a-fa13-43a3-8d81-6191d115ff17-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:48.591885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.591855    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls" pod "0c0a447a-fa13-43a3-8d81-6191d115ff17" (UID: "0c0a447a-fa13-43a3-8d81-6191d115ff17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:48.591885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.591875    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs" pod "0c0a447a-fa13-43a3-8d81-6191d115ff17" (UID: "0c0a447a-fa13-43a3-8d81-6191d115ff17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:06:48.592015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.591901    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn" pod "0c0a447a-fa13-43a3-8d81-6191d115ff17" (UID: "0c0a447a-fa13-43a3-8d81-6191d115ff17"). InnerVolumeSpecName "kube-api-access-hcspn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:06:48.692378 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.692332    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcspn\" (UniqueName: \"kubernetes.io/projected/0c0a447a-fa13-43a3-8d81-6191d115ff17-kube-api-access-hcspn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:48.692378 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.692376    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:48.692594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.692390    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c0a447a-fa13-43a3-8d81-6191d115ff17-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:06:48.988710 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.988674    6931 generic.go:356] "Generic (PLEG): container finished" podID="0c0a447a-fa13-43a3-8d81-6191d115ff17" containerID="46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09" exitCode=0
Oct 31 05:06:48.988906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.988728    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"
Oct 31 05:06:48.988906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.988761    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" event={"ID":"0c0a447a-fa13-43a3-8d81-6191d115ff17","Type":"ContainerDied","Data":"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"}
Oct 31 05:06:48.988906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.988819    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm" event={"ID":"0c0a447a-fa13-43a3-8d81-6191d115ff17","Type":"ContainerDied","Data":"af869de835132ee989ded88e24e2481a2a0b058bd5999f8e21ecc4ca4ef4a10c"}
Oct 31 05:06:48.988906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:48.988841    6931 scope.go:122] "RemoveContainer" containerID="46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"
Oct 31 05:06:49.000178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:49.000145    6931 scope.go:122] "RemoveContainer" containerID="46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"
Oct 31 05:06:49.000534 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:06:49.000475    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09\": container with ID starting with 46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09 not found: ID does not exist" containerID="46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"
Oct 31 05:06:49.000604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:49.000548    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09"} err="failed to get container status \"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09\": rpc error: code = NotFound desc = could not find container \"46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09\": container with ID starting with 46b182def1ea33942913143c597392098be8316bf7c12475f0bb2e88c0606a09 not found: ID does not exist"
Oct 31 05:06:49.039962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:49.039925    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 05:06:49.051880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:49.051848    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb346b-797cbff7d4-7fkfm"]
Oct 31 05:06:49.107924 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:49.107880    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:50.123634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:50.123600    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0a447a-fa13-43a3-8d81-6191d115ff17" path="/var/lib/kubelet/pods/0c0a447a-fa13-43a3-8d81-6191d115ff17/volumes"
Oct 31 05:06:50.616541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:50.616500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:50.867086 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:50.867003    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:52.332880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:52.332840    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:52.557782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:52.557743    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:52.764528 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:52.764441    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:53.156036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:53.155996    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:53.570604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:53.570565    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:53.818196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:53.817970    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:54.027081 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:54.026989    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:54.515600 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:54.515558    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:54.739688 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:54.739648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:54.955172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:54.955125    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:55.174546 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:55.174508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:55.410128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:55.410092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:55.859107 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:55.859076    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:56.774302 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:56.774268    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:57.385223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:57.385179    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:06:58.611653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:06:58.611611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:00.498916 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:00.498879    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:02.225830 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:02.225791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:02.525110 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:02.525020    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:02.825169 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:02.825130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:03.243441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:03.243345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:03.951313 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:03.951279    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 05:07:03.951572 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:03.951544    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerName="maas-api" containerID="cri-o://5993b264bc9df35232bd05c8f9bd450517655e02aab8ffd4efcc9eb5dbc8c609" gracePeriod=30
Oct 31 05:07:04.053666 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.053623    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": dial tcp 10.134.0.169:8443: connect: connection refused"
Oct 31 05:07:04.088369 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.088338    6931 generic.go:356] "Generic (PLEG): container finished" podID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerID="5993b264bc9df35232bd05c8f9bd450517655e02aab8ffd4efcc9eb5dbc8c609" exitCode=0
Oct 31 05:07:04.088538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.088418    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" event={"ID":"14ec7070-97da-4ec2-9261-fdf77c5a08c0","Type":"ContainerDied","Data":"5993b264bc9df35232bd05c8f9bd450517655e02aab8ffd4efcc9eb5dbc8c609"}
Oct 31 05:07:04.120042 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.119991    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:04.297258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.297220    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 05:07:04.387891 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.387853    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\" (UniqueName: \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\") pod \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") "
Oct 31 05:07:04.388072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.387924    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") pod \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") "
Oct 31 05:07:04.388197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.388101    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\") pod \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") "
Oct 31 05:07:04.388197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.388186    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\") pod \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\" (UID: \"14ec7070-97da-4ec2-9261-fdf77c5a08c0\") "
Oct 31 05:07:04.388901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.388875    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca" pod "14ec7070-97da-4ec2-9261-fdf77c5a08c0" (UID: "14ec7070-97da-4ec2-9261-fdf77c5a08c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:07:04.390146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.390121    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls" pod "14ec7070-97da-4ec2-9261-fdf77c5a08c0" (UID: "14ec7070-97da-4ec2-9261-fdf77c5a08c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:04.390286 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.390143    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg" pod "14ec7070-97da-4ec2-9261-fdf77c5a08c0" (UID: "14ec7070-97da-4ec2-9261-fdf77c5a08c0"). InnerVolumeSpecName "kube-api-access-dnpwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:07:04.390514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.390486    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs" pod "14ec7070-97da-4ec2-9261-fdf77c5a08c0" (UID: "14ec7070-97da-4ec2-9261-fdf77c5a08c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:04.490076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.490036    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:04.490076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.490071    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec7070-97da-4ec2-9261-fdf77c5a08c0-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:04.490076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.490081    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpwg\" (UniqueName: \"kubernetes.io/projected/14ec7070-97da-4ec2-9261-fdf77c5a08c0-kube-api-access-dnpwg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:04.490422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.490090    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14ec7070-97da-4ec2-9261-fdf77c5a08c0-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:04.981184 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:04.981101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:05.096419 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.096383    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"
Oct 31 05:07:05.096591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.096426    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk" event={"ID":"14ec7070-97da-4ec2-9261-fdf77c5a08c0","Type":"ContainerDied","Data":"e3fff9417d2e569428ad502a1823a77e756a40cb6c94dcc43089334266102880"}
Oct 31 05:07:05.096591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.096512    6931 scope.go:122] "RemoveContainer" containerID="5993b264bc9df35232bd05c8f9bd450517655e02aab8ffd4efcc9eb5dbc8c609"
Oct 31 05:07:05.147840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.147807    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 05:07:05.159816 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.159785    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c1fe26-5b66bb6974-gq5nk"]
Oct 31 05:07:05.507519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:05.507484    6931 scope.go:122] "RemoveContainer" containerID="63bfe7bd046ab1e439d91c1bf18bdb412458a8ea0e2554aaca9d3f84a35e8043"
Oct 31 05:07:06.125952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:06.125921    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" path="/var/lib/kubelet/pods/14ec7070-97da-4ec2-9261-fdf77c5a08c0/volumes"
Oct 31 05:07:06.701509 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:06.701475    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:08.486495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:08.486456    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:08.707970 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:08.707934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:10.302859 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:10.302811    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:10.759713 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:07:10.759623    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:07:12.100874 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:12.100825    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:13.521772 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:13.521740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:13.777571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:13.777483    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:14.000525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:14.000488    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:14.387620 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:14.387571    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:14.780857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:14.780817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:15.932942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:15.932902    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:17.709206 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:17.709130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:19.440999 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:19.440962    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:20.828498 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:20.828454    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:21.049627 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:21.049589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:21.339424 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:21.339386    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:21.543282 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:21.543244    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:21.957623 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:21.957582    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:22.171430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:22.171389    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:23.085790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:23.085755    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:24.892450 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:24.892409    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:26.727069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:26.727025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:27.644566 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:27.644524    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:28.546475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:28.546437    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:30.272892 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:30.272858    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:31.680676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:31.680637    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 05:07:31.681417 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:31.681385    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" podUID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" containerName="maas-api" containerID="cri-o://19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080" gracePeriod=30
Oct 31 05:07:31.854424 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:31.854377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:32.018178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.017343    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 05:07:32.036340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.035888    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") pod \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") "
Oct 31 05:07:32.036340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.035974    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\") pod \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") "
Oct 31 05:07:32.036340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.036061    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\" (UniqueName: \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\") pod \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") "
Oct 31 05:07:32.037539 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.036149    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\") pod \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\" (UID: \"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd\") "
Oct 31 05:07:32.038173 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.037818    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca" pod "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" (UID: "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:07:32.040448 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.040418    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls" pod "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" (UID: "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:32.043022 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.042986    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs" pod "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" (UID: "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:32.043557 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.043510    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7" pod "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" (UID: "f01e9a3c-c437-484f-a9a4-a1034aaeb9fd"). InnerVolumeSpecName "kube-api-access-8vwx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:07:32.139093 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.139058    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:32.139093 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.139093    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:32.139345 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.139108    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:32.139345 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.139120    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwx7\" (UniqueName: \"kubernetes.io/projected/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd-kube-api-access-8vwx7\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:32.273411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.273312    6931 generic.go:356] "Generic (PLEG): container finished" podID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" containerID="19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080" exitCode=0
Oct 31 05:07:32.273849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.273450    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" event={"ID":"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd","Type":"ContainerDied","Data":"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"}
Oct 31 05:07:32.273849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.273501    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99" event={"ID":"f01e9a3c-c437-484f-a9a4-a1034aaeb9fd","Type":"ContainerDied","Data":"84d116d2ebccf33400808d98a198bd9e2148838133786faee318dbfe1a3be17d"}
Oct 31 05:07:32.273849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.273530    6931 scope.go:122] "RemoveContainer" containerID="19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"
Oct 31 05:07:32.273849 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.273700    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"
Oct 31 05:07:32.286272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.286247    6931 scope.go:122] "RemoveContainer" containerID="19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"
Oct 31 05:07:32.287758 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:07:32.286786    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080\": container with ID starting with 19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080 not found: ID does not exist" containerID="19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"
Oct 31 05:07:32.287758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.286820    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080"} err="failed to get container status \"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080\": rpc error: code = NotFound desc = could not find container \"19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080\": container with ID starting with 19bbb11454208ddb508b6e9560f6a35c778688e0a380ae18cec915d47a0d3080 not found: ID does not exist"
Oct 31 05:07:32.327845 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.327807    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 05:07:32.339561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.339521    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12c0bc-55ddfffdfb-sqf99"]
Oct 31 05:07:32.595500 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:32.595456    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:34.122454 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:34.122418    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" path="/var/lib/kubelet/pods/f01e9a3c-c437-484f-a9a4-a1034aaeb9fd/volumes"
Oct 31 05:07:34.383780 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:34.383697    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:34.587805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:34.587773    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:36.165541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:36.165504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:37.300642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:37.300603    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:37.534683 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:37.534648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:37.911653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:37.911618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:38.121213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:38.121143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:38.540865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:38.540831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:39.490394 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:39.490355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:39.703380 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:39.703343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:40.007347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:40.007305    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:40.767200 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:07:40.767140    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:07:41.306485 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:41.306433    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 05:07:41.307135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:41.307100    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" podUID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" containerName="maas-api" containerID="cri-o://be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003" gracePeriod=30
Oct 31 05:07:41.524201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:41.510557    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:41.924645 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:41.924515    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:42.223112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.223041    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 05:07:42.319397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.319359    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\") pod \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") "
Oct 31 05:07:42.319616 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.319466    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\") pod \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") "
Oct 31 05:07:42.319616 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.319555    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\") pod \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") "
Oct 31 05:07:42.319760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.319625    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\" (UniqueName: \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\") pod \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\" (UID: \"bac27bec-2640-4894-bc1b-4b673c3bc6dc\") "
Oct 31 05:07:42.319866 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.319840    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca" pod "bac27bec-2640-4894-bc1b-4b673c3bc6dc" (UID: "bac27bec-2640-4894-bc1b-4b673c3bc6dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:07:42.321270 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.320190    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac27bec-2640-4894-bc1b-4b673c3bc6dc-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:42.322643 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.322606    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs" pod "bac27bec-2640-4894-bc1b-4b673c3bc6dc" (UID: "bac27bec-2640-4894-bc1b-4b673c3bc6dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:42.322905 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.322873    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx" pod "bac27bec-2640-4894-bc1b-4b673c3bc6dc" (UID: "bac27bec-2640-4894-bc1b-4b673c3bc6dc"). InnerVolumeSpecName "kube-api-access-ppbdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:07:42.323031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.323009    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls" pod "bac27bec-2640-4894-bc1b-4b673c3bc6dc" (UID: "bac27bec-2640-4894-bc1b-4b673c3bc6dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:07:42.423939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.423175    6931 generic.go:356] "Generic (PLEG): container finished" podID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" containerID="be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003" exitCode=0
Oct 31 05:07:42.423939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.423233    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" event={"ID":"bac27bec-2640-4894-bc1b-4b673c3bc6dc","Type":"ContainerDied","Data":"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"}
Oct 31 05:07:42.423939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.423271    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2" event={"ID":"bac27bec-2640-4894-bc1b-4b673c3bc6dc","Type":"ContainerDied","Data":"c44d0bb9dfb8098b0ae09f536a5a5147a9cf87c3ae016874c408de8424c95c59"}
Oct 31 05:07:42.423939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.423294    6931 scope.go:122] "RemoveContainer" containerID="be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"
Oct 31 05:07:42.423939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.423404    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"
Oct 31 05:07:42.424377 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.424227    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:42.424377 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.424246    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbdx\" (UniqueName: \"kubernetes.io/projected/bac27bec-2640-4894-bc1b-4b673c3bc6dc-kube-api-access-ppbdx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:42.424377 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.424257    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac27bec-2640-4894-bc1b-4b673c3bc6dc-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:07:42.519246 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.514344    6931 scope.go:122] "RemoveContainer" containerID="be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"
Oct 31 05:07:42.519246 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:07:42.514749    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003\": container with ID starting with be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003 not found: ID does not exist" containerID="be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"
Oct 31 05:07:42.519246 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.514778    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003"} err="failed to get container status \"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003\": rpc error: code = NotFound desc = could not find container \"be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003\": container with ID starting with be03f309b1d5d118a885816c71d416c72dea83ca82aba09f695de603be30a003 not found: ID does not exist"
Oct 31 05:07:42.609329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.609280    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 05:07:42.611055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.611021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:42.611501 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.611477    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47e9af-59cb46d459-mhgr2"]
Oct 31 05:07:42.939323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.939276    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:07:42.939804 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:42.939350    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:07:43.288884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:43.288825    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:43.508952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:43.508904    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:44.033383 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:44.033343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:44.123112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:44.123077    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" path="/var/lib/kubelet/pods/bac27bec-2640-4894-bc1b-4b673c3bc6dc/volumes"
Oct 31 05:07:44.436816 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:44.436711    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:44.645369 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:44.645310    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:45.342191 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:45.342116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:45.636308 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:45.636219    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:46.616228 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:46.616189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:48.420634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:48.420588    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:50.217953 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:50.217906    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:50.419409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:50.419362    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:51.108726 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:51.108690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:51.993431 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:51.993395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:53.717812 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:53.717777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:55.520346 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:55.520311    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:57.306560 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:57.306523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:57.521659 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:57.521621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:07:59.087489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:07:59.087450    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:00.918372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:00.918319    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:01.719710 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:01.719668    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:03.006716 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:03.006644    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:03.454180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:03.453225    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:04.894667 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:04.894623    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:06.692558 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:06.692518    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:08.120026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:08.119984    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:08.511769 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:08.511684    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:10.300828 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:10.300788    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:10.758959 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:08:10.758865    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:08:12.023000 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:12.022965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:13.917700 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:13.917663    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:15.707093 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:15.707051    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:15.920724 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:15.920683    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:17.559766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:17.559726    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:19.437968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:19.437936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:21.226007 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:21.225971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:21.438689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:21.438646    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:21.853741 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:21.853706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:22.313096 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:22.313061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:23.327533 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:23.327495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:24.837974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:24.837931    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:25.048197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:25.048130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:25.258551 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:25.258429    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:25.895556 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:25.895490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:26.108612 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:26.108568    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:27.219401 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.219369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:27.619611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.619576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:27.716193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716142    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:08:27.716868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716850    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0c0a447a-fa13-43a3-8d81-6191d115ff17" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716870    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0a447a-fa13-43a3-8d81-6191d115ff17" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716880    6931 cpu_manager.go:414] "containerMap: removing container" podUID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716885    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716896    6931 cpu_manager.go:414] "containerMap: removing container" podUID="4e483801-b8a1-44a1-8999-efd4a8faf25d" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716902    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e483801-b8a1-44a1-8999-efd4a8faf25d" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716908    6931 cpu_manager.go:414] "containerMap: removing container" podUID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716913    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" containerName="maas-api"
Oct 31 05:08:27.716925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716925    6931 cpu_manager.go:414] "containerMap: removing container" podUID="529a5853-adf2-4250-96d1-9a9fc9d251e2" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716929    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529a5853-adf2-4250-96d1-9a9fc9d251e2" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716938    6931 cpu_manager.go:414] "containerMap: removing container" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716944    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716956    6931 cpu_manager.go:414] "containerMap: removing container" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716960    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716985    6931 cpu_manager.go:414] "containerMap: removing container" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.716989    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717069    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7f1f45-03a8-4c35-9c8d-c3c9c287742a" containerName="payload-processing"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717083    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0a447a-fa13-43a3-8d81-6191d115ff17" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717089    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01e9a3c-c437-484f-a9a4-a1034aaeb9fd" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717096    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e483801-b8a1-44a1-8999-efd4a8faf25d" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717102    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac27bec-2640-4894-bc1b-4b673c3bc6dc" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717108    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="529a5853-adf2-4250-96d1-9a9fc9d251e2" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717114    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ec7070-97da-4ec2-9261-fdf77c5a08c0" containerName="maas-api"
Oct 31 05:08:27.717254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.717120    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e01560-0500-47bd-8070-4c7e2a50b76c" containerName="payload-pre-processing"
Oct 31 05:08:27.720798 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.720773    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:27.737909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.737882    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:08:27.746896 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.746868    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-51cc4ce6\""
Oct 31 05:08:27.748430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.748395    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-51cc4ce6-dockercfg-4hgbj\""
Oct 31 05:08:27.781962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.781924    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:08:27.786923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.786899    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:27.804517 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.804485    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:08:27.837361 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.837325    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:27.905992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.905900    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:27.905992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.905947    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:27.906224 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.906089    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82jf\" (UniqueName: \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:27.906275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:27.906232    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5kh\" (UniqueName: \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.007277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.007238    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82jf\" (UniqueName: \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.007474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.007341    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5kh\" (UniqueName: \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.007474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.007405    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.007474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.007442    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.008228 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.008204    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.008272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.008216    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.030321 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.030290    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82jf\" (UniqueName: \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\") pod \"payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.030471 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.030391    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5kh\" (UniqueName: \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\") pod \"payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") " pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.034090 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.034070    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.104701 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.103565    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.330788 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.330761    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:08:28.333925 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:08:28.333901    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555dc1e4_6853_45a1_90a1_556242c7563c.slice/crio-46f04d98567a97b81f7a27e94c628a804a92791cf9d150ff0c1b496dcfce7a7d WatchSource:0}: Error finding container 46f04d98567a97b81f7a27e94c628a804a92791cf9d150ff0c1b496dcfce7a7d: Status 404 returned error can't find the container with id 46f04d98567a97b81f7a27e94c628a804a92791cf9d150ff0c1b496dcfce7a7d
Oct 31 05:08:28.413878 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.413852    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:08:28.419276 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:08:28.419248    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa9411_eaaa_4d6f_b9e1_f2e1b1dfa52c.slice/crio-8ba1eca4f6a6c9e1f65c2c5d7b3bfd8c7e697d341caa20fb4f04eecea5005ca7 WatchSource:0}: Error finding container 8ba1eca4f6a6c9e1f65c2c5d7b3bfd8c7e697d341caa20fb4f04eecea5005ca7: Status 404 returned error can't find the container with id 8ba1eca4f6a6c9e1f65c2c5d7b3bfd8c7e697d341caa20fb4f04eecea5005ca7
Oct 31 05:08:28.766791 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.766698    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" event={"ID":"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c","Type":"ContainerStarted","Data":"0f0ba8fc437cc196e9d12120ab9ab2a14f40246c9cb00773713bd72a41fc5482"}
Oct 31 05:08:28.766791 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.766757    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" event={"ID":"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c","Type":"ContainerStarted","Data":"8ba1eca4f6a6c9e1f65c2c5d7b3bfd8c7e697d341caa20fb4f04eecea5005ca7"}
Oct 31 05:08:28.767027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.766902    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:28.768301 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.768274    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" event={"ID":"555dc1e4-6853-45a1-90a1-556242c7563c","Type":"ContainerStarted","Data":"279e68f31d3583bcd65ae5be0fb486d8894c37ca3ea889f345706363166bc8b5"}
Oct 31 05:08:28.768411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.768311    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" event={"ID":"555dc1e4-6853-45a1-90a1-556242c7563c","Type":"ContainerStarted","Data":"46f04d98567a97b81f7a27e94c628a804a92791cf9d150ff0c1b496dcfce7a7d"}
Oct 31 05:08:28.768411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.768363    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:28.808588 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.808499    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" podStartSLOduration=1.808481561 podStartE2EDuration="1.808481561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:08:28.806793402 +0000 UTC m=+3804.839793101" watchObservedRunningTime="2026-10-31 05:08:28.808481561 +0000 UTC m=+3804.841481240"
Oct 31 05:08:28.846470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:28.846413    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" podStartSLOduration=1.846397847 podStartE2EDuration="1.846397847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:08:28.845530727 +0000 UTC m=+3804.878530407" watchObservedRunningTime="2026-10-31 05:08:28.846397847 +0000 UTC m=+3804.879397526"
Oct 31 05:08:29.028556 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:29.028472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:30.912576 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:30.912538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:32.643975 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:32.643934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:34.412492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:34.412451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:36.211892 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:36.211842    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:38.001666 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:38.001624    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:39.734704 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:39.734660    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:39.778864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:39.778835    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:08:39.779039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:39.778901    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:08:40.760112 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:08:40.760076    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:08:41.506102 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:41.506067    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:43.230870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:43.230831    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:44.948959 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:44.948920    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:46.707207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:46.707145    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:48.429805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:48.429762    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:50.227839 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:50.227806    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:51.977281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:51.977239    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:52.186141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:52.186100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:52.389867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:52.389828    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:54.013883 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:54.013822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:55.812118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:55.812080    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:57.615371 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:57.615333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:58.454584 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:58.454552    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:58.664950 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:58.664914    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:08:59.427883 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:08:59.427851    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:01.034283 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.034249    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:09:01.039018 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.038995    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.067187 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.067142    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-766728\"/\"kube-root-ca.crt\""
Oct 31 05:09:01.067395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.067225    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-766728\"/\"openshift-service-ca.crt\""
Oct 31 05:09:01.067395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.067254    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-766728\"/\"default-dockercfg-q5f27\""
Oct 31 05:09:01.067519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.067408    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-766728\"/\"sub-test-model-766728-kserve-self-signed-certs\""
Oct 31 05:09:01.069820 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.069762    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:09:01.086778 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.086742    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.086967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.086938    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgf2d\" (UniqueName: \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.087041 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.087023    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.087104 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.087077    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.087186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.087111    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.087252 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.087204    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.188719 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.188674    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgf2d\" (UniqueName: \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.188940 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.188730    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.188940 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.188908    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189071 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.188949    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189071 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.188989    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189071 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.189066    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189251 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.189107    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189339 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.189308    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.189412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.189394    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.191679 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.191640    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.192035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.192011    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.211899 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.211868    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgf2d\" (UniqueName: \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\") pod \"sub-test-model-766728-kserve-6f7d896bfc-qhxcl\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") " pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.219125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.219077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:01.355825 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.355738    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:01.468454 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.468421    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:01.660016 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.659933    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:09:01.663430 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:09:01.663406    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc0203d3_e103_4d9a_a98c_ff68b4606859.slice/crio-983e4b88a19e4d3d6e4de5c01ba76341245bd02eeac979b096f2849df7f47c09 WatchSource:0}: Error finding container 983e4b88a19e4d3d6e4de5c01ba76341245bd02eeac979b096f2849df7f47c09: Status 404 returned error can't find the container with id 983e4b88a19e4d3d6e4de5c01ba76341245bd02eeac979b096f2849df7f47c09
Oct 31 05:09:01.981468 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.981373    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" event={"ID":"bc0203d3-e103-4d9a-a98c-ff68b4606859","Type":"ContainerStarted","Data":"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"}
Oct 31 05:09:01.981468 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:01.981425    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" event={"ID":"bc0203d3-e103-4d9a-a98c-ff68b4606859","Type":"ContainerStarted","Data":"983e4b88a19e4d3d6e4de5c01ba76341245bd02eeac979b096f2849df7f47c09"}
Oct 31 05:09:02.023588 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:02.023534    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" podStartSLOduration=1.023518387 podStartE2EDuration="1.023518387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:09:02.021291265 +0000 UTC m=+3838.054290945" watchObservedRunningTime="2026-10-31 05:09:02.023518387 +0000 UTC m=+3838.056518065"
Oct 31 05:09:02.356697 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:02.356662    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:03.157455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:03.157413    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:03.578550 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:03.578517    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:05.001624 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:05.001588    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:06.790290 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:06.790250    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:08.635038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:08.634999    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:10.475760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:10.475716    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:10.759859 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:09:10.759773    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:09:11.356236 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:11.356197    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:11.369476 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:11.369452    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:11.979489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:11.979442    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:09:11.983503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:11.983484    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.012484 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.012449    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-51cc4ce6-serving-cert\""
Oct 31 05:09:12.017069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.017030    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:09:12.056537 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.056505    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:09:12.093353 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.093316    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:12.119691 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.119651    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.119911 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.119718    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.120070 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.120044    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.120320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.120296    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vhll\" (UniqueName: \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.221368 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.221331    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.221594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.221391    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.221594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.221521    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.221594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.221562    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vhll\" (UniqueName: \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.222265 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.222237    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.224107 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.224082    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.224249 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.224231    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.245414 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.245332    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vhll\" (UniqueName: \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\") pod \"maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.296744 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.296712    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:12.308933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.308897    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:12.522330 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.522224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:12.601569 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.601533    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:09:12.604874 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:09:12.604846    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9add392e_bf73_47f3_83e5_878729b7b21a.slice/crio-f90e23f3ee3f74cb038e7c0941e8c7b880e371786d60b6b85787e07ea9741833 WatchSource:0}: Error finding container f90e23f3ee3f74cb038e7c0941e8c7b880e371786d60b6b85787e07ea9741833: Status 404 returned error can't find the container with id f90e23f3ee3f74cb038e7c0941e8c7b880e371786d60b6b85787e07ea9741833
Oct 31 05:09:12.606294 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:12.606279    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 05:09:13.055713 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:13.055679    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" event={"ID":"9add392e-bf73-47f3-83e5-878729b7b21a","Type":"ContainerStarted","Data":"f90e23f3ee3f74cb038e7c0941e8c7b880e371786d60b6b85787e07ea9741833"}
Oct 31 05:09:14.063861 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:14.063827    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" event={"ID":"9add392e-bf73-47f3-83e5-878729b7b21a","Type":"ContainerStarted","Data":"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"}
Oct 31 05:09:14.064317 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:14.063944    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:14.107489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:14.107434    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" podStartSLOduration=2.704497059 podStartE2EDuration="3.107420339s" totalImagesPullingTime="402.92328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:09:14.10464511 +0000 UTC m=+3850.137644789" watchObservedRunningTime="2026-10-31 05:09:14.107420339 +0000 UTC m=+3850.140420019"
Oct 31 05:09:14.222004 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:14.221960    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:14.439027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:14.438908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:15.778338 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:15.778290    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:15.998928 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:15.998886    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:17.897027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:17.896980    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:19.719699 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:19.719638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:20.075321 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:20.075289    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:09:21.798642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:21.798599    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:23.031304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:23.031267    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:23.697707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:23.697666    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:25.503082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:25.503039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:26.255820 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.255779    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:09:26.261323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.261291    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.281515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.281486    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:09:26.288847 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.288821    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-25f56e\"/\"default-dockercfg-gbpkr\""
Oct 31 05:09:26.288982 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.288900    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-25f56e\"/\"sub-test-model-25f56e-kserve-self-signed-certs\""
Oct 31 05:09:26.291470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.291452    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-25f56e\"/\"kube-root-ca.crt\""
Oct 31 05:09:26.291470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.291456    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-25f56e\"/\"openshift-service-ca.crt\""
Oct 31 05:09:26.294232 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294211    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.294335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294268    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.294386 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294370    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.294501 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294483    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.294550 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294521    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6qn\" (UniqueName: \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.294619 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.294603    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396292 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396248    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396477 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396322    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396477 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396348    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396636 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396500    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396636 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396541    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6qn\" (UniqueName: \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.396884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396826    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.397015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396886    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.397015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.396622    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.397227 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.397097    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.398883 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.398852    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.398980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.398938    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.410659 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.410623    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:26.420290 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.420261    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6qn\" (UniqueName: \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\") pod \"sub-test-model-25f56e-kserve-5b4cd89879-2d5gl\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") " pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.579016 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.578974    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:26.883531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:26.883505    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:09:26.886446 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:09:26.886420    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod425452e2_5ddf_4958_8eb1_d4fd1ebba711.slice/crio-9216dce07b339c56d9babfbf4fc62c68acb3b41bc69c99b6efe9d7de2280add1 WatchSource:0}: Error finding container 9216dce07b339c56d9babfbf4fc62c68acb3b41bc69c99b6efe9d7de2280add1: Status 404 returned error can't find the container with id 9216dce07b339c56d9babfbf4fc62c68acb3b41bc69c99b6efe9d7de2280add1
Oct 31 05:09:27.144764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:27.144669    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" event={"ID":"425452e2-5ddf-4958-8eb1-d4fd1ebba711","Type":"ContainerStarted","Data":"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"}
Oct 31 05:09:27.144764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:27.144721    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" event={"ID":"425452e2-5ddf-4958-8eb1-d4fd1ebba711","Type":"ContainerStarted","Data":"9216dce07b339c56d9babfbf4fc62c68acb3b41bc69c99b6efe9d7de2280add1"}
Oct 31 05:09:27.186738 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:27.186681    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" podStartSLOduration=1.186665488 podStartE2EDuration="1.186665488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:09:27.184564035 +0000 UTC m=+3863.217563736" watchObservedRunningTime="2026-10-31 05:09:27.186665488 +0000 UTC m=+3863.219665167"
Oct 31 05:09:27.540338 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:27.540297    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:27.579101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:27.579067    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:29.305772 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:29.305735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:30.996190 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:30.996132    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:32.828037 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:32.828001    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:34.695230 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:34.695182    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:34.902332 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:34.902291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:36.494725 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:36.494685    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:36.579626 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:36.579592    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:36.593076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:36.593046    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:37.217678 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:37.217639    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:09:38.226706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:38.226670    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:40.091802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:40.091759    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:40.760275 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:09:40.760239    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:09:41.860439 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:41.860403    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:43.743895 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:43.743736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:45.814594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:45.814505    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:47.727860 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:47.727813    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:49.419210 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.419169    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:09:49.424222 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.424185    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.452376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.452342    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"openshift-service-ca.crt\""
Oct 31 05:09:49.452376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.452374    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"kube-root-ca.crt\""
Oct 31 05:09:49.452591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.452469    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"rate-test-model-57dc6d-kserve-self-signed-certs\""
Oct 31 05:09:49.452591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.452471    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-57dc6d\"/\"default-dockercfg-4r69d\""
Oct 31 05:09:49.457043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.457011    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:09:49.516674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.516632    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:49.528884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.528845    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qllp\" (UniqueName: \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.529069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.528903    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.529148 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.529062    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.529148 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.529123    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.529301 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.529247    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.529359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.529347    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.630779 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.630742    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qllp\" (UniqueName: \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.630980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.630795    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.630980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.630865    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.630980 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.630905    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.631171 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.630985    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.631171 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.631087    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.631401 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.631380    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.631475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.631440    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.631540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.631484    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.633435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.633411    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.633882 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.633857    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.653859 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.653827    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qllp\" (UniqueName: \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\") pod \"rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") " pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.740204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.740097    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:49.820472 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:49.820436    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:50.060790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:50.060723    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:09:50.064825 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:09:50.064799    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21c89fd6_7cb1_4216_b0db_3d0811971743.slice/crio-ebbd8b2fc6addf25057f3fb9bb10712b7eccecebf5f3cdddb2cdfe52f368b9e8 WatchSource:0}: Error finding container ebbd8b2fc6addf25057f3fb9bb10712b7eccecebf5f3cdddb2cdfe52f368b9e8: Status 404 returned error can't find the container with id ebbd8b2fc6addf25057f3fb9bb10712b7eccecebf5f3cdddb2cdfe52f368b9e8
Oct 31 05:09:50.318190 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:50.318087    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" event={"ID":"21c89fd6-7cb1-4216-b0db-3d0811971743","Type":"ContainerStarted","Data":"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"}
Oct 31 05:09:50.318190 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:50.318135    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" event={"ID":"21c89fd6-7cb1-4216-b0db-3d0811971743","Type":"ContainerStarted","Data":"ebbd8b2fc6addf25057f3fb9bb10712b7eccecebf5f3cdddb2cdfe52f368b9e8"}
Oct 31 05:09:50.359702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:50.359645    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" podStartSLOduration=1.359630739 podStartE2EDuration="1.359630739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:09:50.359097779 +0000 UTC m=+3886.392097476" watchObservedRunningTime="2026-10-31 05:09:50.359630739 +0000 UTC m=+3886.392630420"
Oct 31 05:09:50.740717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:50.740635    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:51.411114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:51.411072    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:53.300784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:53.300738    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:55.038702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:55.038658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:56.829374 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:56.829330    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:58.612414 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:58.612374    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:09:59.741230 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:59.741193    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:09:59.753716 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:09:59.753692    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:10:00.393187 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:00.393135    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:10:00.421806 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:00.421771    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:02.135439 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:02.135399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:03.699415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:03.699378    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:03.975687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:03.975607    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:05.542589 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.542546    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:10:05.548257 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.548235    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.581570 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.581542    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"openshift-service-ca.crt\""
Oct 31 05:10:05.581856 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.581828    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"default-dockercfg-bwbnz\""
Oct 31 05:10:05.582056 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.581835    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"kube-root-ca.crt\""
Oct 31 05:10:05.582205 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.582016    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9633f0\"/\"rate-test-model-9633f0-kserve-self-signed-certs\""
Oct 31 05:10:05.584873 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.584850    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:10:05.636721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.636673    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtk9v\" (UniqueName: \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.636938 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.636876    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.637035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.637006    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.637197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.637075    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.637197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.637121    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.637304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.637247    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.738743 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738700    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.738956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738759    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.738956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738785    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.738956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738828    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.738956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738864    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtk9v\" (UniqueName: \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.739224 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.738968    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.739287 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.739225    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.739331 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.739287    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.739612 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.739583    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.741785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.741756    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.742039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.742022    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.761735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.761696    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:05.763197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.763171    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtk9v\" (UniqueName: \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\") pod \"rate-test-model-9633f0-kserve-7cd47f5969-mrcnr\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") " pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:05.863481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:05.863382    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:06.003705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.003665    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:06.171844 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.171817    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:10:06.174619 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:10:06.174584    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2cb2c0_5771_4d5d_9a1c_20a978f44b03.slice/crio-e8bbe10f638e0cc4a4de349ac69ce7dd0d208e9ee0ab2564519f90c959add884 WatchSource:0}: Error finding container e8bbe10f638e0cc4a4de349ac69ce7dd0d208e9ee0ab2564519f90c959add884: Status 404 returned error can't find the container with id e8bbe10f638e0cc4a4de349ac69ce7dd0d208e9ee0ab2564519f90c959add884
Oct 31 05:10:06.423063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.422968    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" event={"ID":"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03","Type":"ContainerStarted","Data":"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"}
Oct 31 05:10:06.423063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.423013    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" event={"ID":"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03","Type":"ContainerStarted","Data":"e8bbe10f638e0cc4a4de349ac69ce7dd0d208e9ee0ab2564519f90c959add884"}
Oct 31 05:10:06.469912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.469844    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" podStartSLOduration=1.469824872 podStartE2EDuration="1.469824872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:10:06.468208343 +0000 UTC m=+3902.501208015" watchObservedRunningTime="2026-10-31 05:10:06.469824872 +0000 UTC m=+3902.502824556"
Oct 31 05:10:06.864403 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:06.864364    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:07.894171 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:07.894126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:08.307101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:08.307066    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:09.123901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:09.123862    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:09.335545 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:09.335500    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:09.748399 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:09.748358    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:09.975384 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:09.975346    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:10.759822 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:10:10.759782    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:10:11.611701 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:11.611658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:13.372935 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:13.372901    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:15.150487 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:15.150449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:15.863917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:15.863883    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:15.877173 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:15.877132    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:15.910015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:15.909975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:16.500999 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:16.500967    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:10:16.974095 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:16.974061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:18.901876 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:18.901835    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:20.644937 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:20.644899    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:22.524536 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:22.524477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:24.410117 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:24.410075    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:26.253940 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:26.253904    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:27.973389 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:27.973351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:29.823282 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:29.823246    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:31.564283 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:31.564224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:33.294389 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:33.294350    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:34.837090 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:34.837053    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:35.211878 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:35.211773    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:37.006434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:37.006399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:38.837893 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:38.837859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:40.738406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:40.738372    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:40.765790 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:10:40.765753    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:10:40.952917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:40.952881    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:41.208229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:41.208188    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:42.698694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:42.698649    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:44.482934 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:44.482897    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:46.269235 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:46.269200    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:46.500135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:46.500102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:46.715292 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:46.715255    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:47.836807 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:47.836763    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:48.311523 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:48.311484    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:50.096530 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:50.096469    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:51.832395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:51.832357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:52.827352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:52.827290    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:53.039545 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:53.039510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:53.372790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:53.372757    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:53.747445 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:53.747351    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:55.508878 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:55.508827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:57.258137 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:57.258098    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:58.980885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:58.980836    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:59.190706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:59.190653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:10:59.396910 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:10:59.396863    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:01.014205 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:01.014167    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:02.876244 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:02.876210    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:03.816734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:03.816694    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:04.672419 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:04.672377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:05.580857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:05.580822    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:06.509423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:06.509381    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:08.318981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:08.318944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:10.044912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:10.044877    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:10.760049 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:11:10.760007    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:11:11.816805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:11.816767    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:13.611561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:13.611523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:15.390450 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:15.390362    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:16.788759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:16.788724    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 05:11:16.789837 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:16.788962    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" podUID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" containerName="maas-api" containerID="cri-o://2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e" gracePeriod=30
Oct 31 05:11:16.958457 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:16.958408    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:17.119891 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.119850    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 05:11:17.185958 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.185921    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\") pod \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") "
Oct 31 05:11:17.186179 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.186040    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\") pod \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") "
Oct 31 05:11:17.186179 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.186132    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") pod \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") "
Oct 31 05:11:17.186335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.186200    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\" (UniqueName: \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\") pod \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\" (UID: \"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0\") "
Oct 31 05:11:17.186554 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.186528    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca" pod "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" (UID: "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:11:17.186754 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.186732    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:17.189344 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.189317    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls" pod "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" (UID: "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:11:17.189435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.189346    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs" pod "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" (UID: "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:11:17.190072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.190048    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6" pod "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" (UID: "98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0"). InnerVolumeSpecName "kube-api-access-p84g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:11:17.288665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.288625    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:17.288665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.288655    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:17.288665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.288666    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84g6\" (UniqueName: \"kubernetes.io/projected/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0-kube-api-access-p84g6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:17.671482 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.671434    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:17.890022 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.889985    6931 generic.go:356] "Generic (PLEG): container finished" podID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" containerID="2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e" exitCode=0
Oct 31 05:11:17.890495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.890028    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" event={"ID":"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0","Type":"ContainerDied","Data":"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"}
Oct 31 05:11:17.890495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.890036    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"
Oct 31 05:11:17.890495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.890069    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f" event={"ID":"98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0","Type":"ContainerDied","Data":"50547864f2b9591067550bfc6ea6b55b763952c027110b3cddfbcda30a1dd963"}
Oct 31 05:11:17.890495 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.890092    6931 scope.go:122] "RemoveContainer" containerID="2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"
Oct 31 05:11:17.904983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.904958    6931 scope.go:122] "RemoveContainer" containerID="2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"
Oct 31 05:11:17.905524 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:11:17.905498    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e\": container with ID starting with 2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e not found: ID does not exist" containerID="2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"
Oct 31 05:11:17.905619 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.905535    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e"} err="failed to get container status \"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e\": rpc error: code = NotFound desc = could not find container \"2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e\": container with ID starting with 2bf4c65386799e5e3e1cf6c99ecb6e130d5a2c57cad4bf7e54999f07cafb962e not found: ID does not exist"
Oct 31 05:11:17.941651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.941560    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 05:11:17.954939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:17.954902    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-47c409-59f8bc6f57-gt89f"]
Oct 31 05:11:18.123256 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:18.123218    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" path="/var/lib/kubelet/pods/98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0/volumes"
Oct 31 05:11:19.254089 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:19.254050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:20.195102 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:20.195059    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:20.988453 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:20.988422    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:21.227263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:21.227229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:21.789291 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:21.789258    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:22.131352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:22.131258    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:22.340676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:22.340637    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:23.038126 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:23.038088    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:24.070555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:24.070508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:24.359801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:24.359697    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:24.575211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:24.575165    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:25.147619 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:25.147584    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:25.508918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:25.508708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:26.219441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:26.219401    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:26.438622 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:26.438574    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:27.995877 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:27.995834    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:28.783523 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:28.783483    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:28.998372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:28.998324    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:29.586571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:29.586529    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:29.938587 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:29.938488    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:30.275886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:30.275782    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:30.487957 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:30.487915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:31.752532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:31.752488    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:32.525627 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:32.525586    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:32.616363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:32.616322    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 05:11:32.724952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:32.724912    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 05:11:32.932051 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:32.932014    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:33.147087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:33.147046    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:33.358823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:33.358783    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:34.559766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:34.559725    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:35.788573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:35.788536    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:35.994454 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:35.994412    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:36.464608 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:36.464564    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:36.678901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:36.678866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:36.992639 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:36.992550    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:37.616685 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:37.616625    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing" containerID="cri-o://24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f" gracePeriod=25
Oct 31 05:11:37.725578 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:37.725534    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing" containerID="cri-o://3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017" gracePeriod=25
Oct 31 05:11:37.761397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:37.761359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:38.533884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:38.533849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:39.264904 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:39.264865    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:40.285973 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:40.285939    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:40.496966 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:40.496930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:40.759899 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:11:40.759812    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:11:41.132623 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:41.132580    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:11:41.132801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:41.132649    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:11:42.048398 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:42.048357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:43.920687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:43.920646    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:45.063705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:45.063652    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:45.271550 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:45.271511    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:45.477199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:45.477104    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:46.014083 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:46.014039    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:46.935322 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:46.935284    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:11:46.965783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:46.965748    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:11:47.078552 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:47.078510    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:47.298013 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:47.297975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:48.126278 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:48.126242    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:49.869445 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:49.869406    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:51.132077 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.132035    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:11:51.132489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.132074    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 05:11:51.242738 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.242695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:51.631327 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.631289    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:51.936191 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.936073    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" podUID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" containerName="payload-processing" containerID="cri-o://0f0ba8fc437cc196e9d12120ab9ab2a14f40246c9cb00773713bd72a41fc5482" gracePeriod=25
Oct 31 05:11:51.966843 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:51.966808    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" podUID="555dc1e4-6853-45a1-90a1-556242c7563c" containerName="payload-pre-processing" containerID="cri-o://279e68f31d3583bcd65ae5be0fb486d8894c37ca3ea889f345706363166bc8b5" gracePeriod=25
Oct 31 05:11:52.109543 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.109509    6931 generic.go:356] "Generic (PLEG): container finished" podID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" containerID="0f0ba8fc437cc196e9d12120ab9ab2a14f40246c9cb00773713bd72a41fc5482" exitCode=0
Oct 31 05:11:52.109718 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.109590    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" event={"ID":"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c","Type":"ContainerDied","Data":"0f0ba8fc437cc196e9d12120ab9ab2a14f40246c9cb00773713bd72a41fc5482"}
Oct 31 05:11:52.112099 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.112065    6931 generic.go:356] "Generic (PLEG): container finished" podID="555dc1e4-6853-45a1-90a1-556242c7563c" containerID="279e68f31d3583bcd65ae5be0fb486d8894c37ca3ea889f345706363166bc8b5" exitCode=0
Oct 31 05:11:52.112272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.112111    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" event={"ID":"555dc1e4-6853-45a1-90a1-556242c7563c","Type":"ContainerDied","Data":"279e68f31d3583bcd65ae5be0fb486d8894c37ca3ea889f345706363166bc8b5"}
Oct 31 05:11:52.246273 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.246246    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:11:52.275670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.275645    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:11:52.341992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.341954    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\" (UniqueName: \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\") pod \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") "
Oct 31 05:11:52.342182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.342074    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\") pod \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\" (UID: \"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c\") "
Oct 31 05:11:52.342477 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.342455    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume" pod "5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" (UID: "5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:11:52.342752 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.342729    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:52.344297 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.344267    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh" pod "5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" (UID: "5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c"). InnerVolumeSpecName "kube-api-access-5w5kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:11:52.444186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.444081    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\" (UniqueName: \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\") pod \"555dc1e4-6853-45a1-90a1-556242c7563c\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") "
Oct 31 05:11:52.444186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.444152    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\") pod \"555dc1e4-6853-45a1-90a1-556242c7563c\" (UID: \"555dc1e4-6853-45a1-90a1-556242c7563c\") "
Oct 31 05:11:52.444503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.444485    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5kh\" (UniqueName: \"kubernetes.io/projected/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c-kube-api-access-5w5kh\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:52.444574 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.444536    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume" pod "555dc1e4-6853-45a1-90a1-556242c7563c" (UID: "555dc1e4-6853-45a1-90a1-556242c7563c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:11:52.446332 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.446304    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf" pod "555dc1e4-6853-45a1-90a1-556242c7563c" (UID: "555dc1e4-6853-45a1-90a1-556242c7563c"). InnerVolumeSpecName "kube-api-access-p82jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:11:52.545559 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.545523    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/555dc1e4-6853-45a1-90a1-556242c7563c-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:52.545559 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:52.545554    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p82jf\" (UniqueName: \"kubernetes.io/projected/555dc1e4-6853-45a1-90a1-556242c7563c-kube-api-access-p82jf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:53.111967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.111931    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:53.119365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.119340    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"
Oct 31 05:11:53.119365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.119347    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv" event={"ID":"5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c","Type":"ContainerDied","Data":"8ba1eca4f6a6c9e1f65c2c5d7b3bfd8c7e697d341caa20fb4f04eecea5005ca7"}
Oct 31 05:11:53.119614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.119410    6931 scope.go:122] "RemoveContainer" containerID="0f0ba8fc437cc196e9d12120ab9ab2a14f40246c9cb00773713bd72a41fc5482"
Oct 31 05:11:53.120811 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.120790    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7" event={"ID":"555dc1e4-6853-45a1-90a1-556242c7563c","Type":"ContainerDied","Data":"46f04d98567a97b81f7a27e94c628a804a92791cf9d150ff0c1b496dcfce7a7d"}
Oct 31 05:11:53.120898 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.120836    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"
Oct 31 05:11:53.132992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.132969    6931 scope.go:122] "RemoveContainer" containerID="279e68f31d3583bcd65ae5be0fb486d8894c37ca3ea889f345706363166bc8b5"
Oct 31 05:11:53.184386 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.184344    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:11:53.196271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.196228    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-51cc4ce6-cb9cdcf4c-x76x7"]
Oct 31 05:11:53.221034 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.220927    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:11:53.223341 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.223320    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-51cc4ce6-68c544d767-fgsbv"]
Oct 31 05:11:53.501766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:53.501679    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:54.123443 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:54.123408    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555dc1e4-6853-45a1-90a1-556242c7563c" path="/var/lib/kubelet/pods/555dc1e4-6853-45a1-90a1-556242c7563c/volumes"
Oct 31 05:11:54.123837 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:54.123823    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" path="/var/lib/kubelet/pods/5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c/volumes"
Oct 31 05:11:55.242992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:55.242953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:57.029865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.029827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:11:57.889865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.889799    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 05:11:57.914769 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.914732    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\" (UniqueName: \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\") pod \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") "
Oct 31 05:11:57.914925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.914838    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\") pod \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\" (UID: \"82010fd1-72b3-46f4-9ec8-f9909ade03b8\") "
Oct 31 05:11:57.915332 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.915301    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume" pod "82010fd1-72b3-46f4-9ec8-f9909ade03b8" (UID: "82010fd1-72b3-46f4-9ec8-f9909ade03b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:11:57.915510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.915481    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82010fd1-72b3-46f4-9ec8-f9909ade03b8-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:57.917112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.917082    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj" pod "82010fd1-72b3-46f4-9ec8-f9909ade03b8" (UID: "82010fd1-72b3-46f4-9ec8-f9909ade03b8"). InnerVolumeSpecName "kube-api-access-25vbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:11:57.975721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:57.975692    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 05:11:58.016724 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.016688    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\" (UniqueName: \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\") pod \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") "
Oct 31 05:11:58.016922 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.016750    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\") pod \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\" (UID: \"f9e33b65-8e7f-4ce6-8199-8c60671a84e4\") "
Oct 31 05:11:58.017210 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.017144    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume" pod "f9e33b65-8e7f-4ce6-8199-8c60671a84e4" (UID: "f9e33b65-8e7f-4ce6-8199-8c60671a84e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:11:58.017423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.017386    6931 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-plugins-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:58.017423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.017416    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25vbj\" (UniqueName: \"kubernetes.io/projected/82010fd1-72b3-46f4-9ec8-f9909ade03b8-kube-api-access-25vbj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:58.019659 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.019630    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m" pod "f9e33b65-8e7f-4ce6-8199-8c60671a84e4" (UID: "f9e33b65-8e7f-4ce6-8199-8c60671a84e4"). InnerVolumeSpecName "kube-api-access-ncc6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:11:58.118198 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.118150    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncc6m\" (UniqueName: \"kubernetes.io/projected/f9e33b65-8e7f-4ce6-8199-8c60671a84e4-kube-api-access-ncc6m\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:11:58.165151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.165072    6931 generic.go:356] "Generic (PLEG): container finished" podID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerID="24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f" exitCode=0
Oct 31 05:11:58.165151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.165120    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"
Oct 31 05:11:58.165359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.165149    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" event={"ID":"82010fd1-72b3-46f4-9ec8-f9909ade03b8","Type":"ContainerDied","Data":"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"}
Oct 31 05:11:58.165359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.165234    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95" event={"ID":"82010fd1-72b3-46f4-9ec8-f9909ade03b8","Type":"ContainerDied","Data":"cbcbef4e9f7c3993e5c0ab8af9d44966b271acded35fff9c90b4ac8be7638554"}
Oct 31 05:11:58.165359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.165257    6931 scope.go:122] "RemoveContainer" containerID="24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"
Oct 31 05:11:58.167031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.167002    6931 generic.go:356] "Generic (PLEG): container finished" podID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerID="3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017" exitCode=0
Oct 31 05:11:58.167124 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.167048    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"
Oct 31 05:11:58.167228 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.167049    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" event={"ID":"f9e33b65-8e7f-4ce6-8199-8c60671a84e4","Type":"ContainerDied","Data":"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"}
Oct 31 05:11:58.167228 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.167200    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll" event={"ID":"f9e33b65-8e7f-4ce6-8199-8c60671a84e4","Type":"ContainerDied","Data":"e419b90b1b7faff4b7b556477be171c14dd583e37b3d0100d4decb55628fb9c1"}
Oct 31 05:11:58.176913 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.176888    6931 scope.go:122] "RemoveContainer" containerID="24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"
Oct 31 05:11:58.177204 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:11:58.177182    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f\": container with ID starting with 24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f not found: ID does not exist" containerID="24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"
Oct 31 05:11:58.177294 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.177216    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f"} err="failed to get container status \"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f\": rpc error: code = NotFound desc = could not find container \"24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f\": container with ID starting with 24f08c25a6bbf006e641babaec8012963509d261b48d21d5265de368c0171d7f not found: ID does not exist"
Oct 31 05:11:58.177294 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.177250    6931 scope.go:122] "RemoveContainer" containerID="3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"
Oct 31 05:11:58.185993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.185965    6931 scope.go:122] "RemoveContainer" containerID="3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"
Oct 31 05:11:58.186308 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:11:58.186287    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017\": container with ID starting with 3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017 not found: ID does not exist" containerID="3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"
Oct 31 05:11:58.186375 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.186315    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017"} err="failed to get container status \"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017\": rpc error: code = NotFound desc = could not find container \"3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017\": container with ID starting with 3d750d0705198306fb6fdf8b1c298b73f6f198d519975928bbbbceb59b89e017 not found: ID does not exist"
Oct 31 05:11:58.215279 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.215243    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 05:11:58.221531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.221505    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a04e4e-566bc9db7f-htzll"]
Oct 31 05:11:58.254008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.253975    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 05:11:58.258111 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.258083    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a04e4e-7f98ff8fc5-ltn95"]
Oct 31 05:11:58.953480 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:11:58.953395    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:00.122965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:00.122930    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" path="/var/lib/kubelet/pods/82010fd1-72b3-46f4-9ec8-f9909ade03b8/volumes"
Oct 31 05:12:00.123449 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:00.123433    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" path="/var/lib/kubelet/pods/f9e33b65-8e7f-4ce6-8199-8c60671a84e4/volumes"
Oct 31 05:12:00.787278 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:00.787234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:02.571828 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:02.571788    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:03.941198 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:03.941146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:04.307058 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:04.307020    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:05.851698 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:05.851654    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:06.725764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:06.725728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:08.610492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:08.610451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:10.391012 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:10.390965    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:10.759421 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:12:10.759329    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:12:11.776095 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:11.776060    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 05:12:11.776540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:11.776279    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" podUID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" containerName="maas-api" containerID="cri-o://881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4" gracePeriod=30
Oct 31 05:12:11.958782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:11.958736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:12.069175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.069137    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 05:12:12.193535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.193496    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\" (UniqueName: \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\") pod \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") "
Oct 31 05:12:12.193733 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.193562    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\") pod \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") "
Oct 31 05:12:12.193733 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.193599    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\") pod \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") "
Oct 31 05:12:12.194076 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.193835    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\") pod \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\" (UID: \"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a\") "
Oct 31 05:12:12.194423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.194381    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca" pod "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" (UID: "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:12:12.196021 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.195987    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls" pod "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" (UID: "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:12.196479 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.196444    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs" pod "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" (UID: "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:12.196565 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.196548    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4" pod "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" (UID: "5acf8b09-b2d2-4a36-a1e0-d438a334ad9a"). InnerVolumeSpecName "kube-api-access-sfsf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:12:12.256771 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.256729    6931 generic.go:356] "Generic (PLEG): container finished" podID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" containerID="881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4" exitCode=0
Oct 31 05:12:12.256949 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.256919    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" event={"ID":"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a","Type":"ContainerDied","Data":"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"}
Oct 31 05:12:12.256994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.256970    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79" event={"ID":"5acf8b09-b2d2-4a36-a1e0-d438a334ad9a","Type":"ContainerDied","Data":"35fa961a496aeb647f66804ee0d24e8042fbbd113ac734b575941c154eb94321"}
Oct 31 05:12:12.257038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.257002    6931 scope.go:122] "RemoveContainer" containerID="881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"
Oct 31 05:12:12.257184 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.257151    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"
Oct 31 05:12:12.295660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.295629    6931 scope.go:122] "RemoveContainer" containerID="881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"
Oct 31 05:12:12.296581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.296243    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:12.296581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.296269    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsf4\" (UniqueName: \"kubernetes.io/projected/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-kube-api-access-sfsf4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:12.296581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.296284    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:12.296581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.296297    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:12.296913 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:12:12.296887    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4\": container with ID starting with 881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4 not found: ID does not exist" containerID="881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"
Oct 31 05:12:12.296986 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.296926    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4"} err="failed to get container status \"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4\": rpc error: code = NotFound desc = could not find container \"881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4\": container with ID starting with 881efb1fb3845d8811c4383490aa2c9b485b9c551b33c7d595cab020022c01d4 not found: ID does not exist"
Oct 31 05:12:12.325309 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.325275    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 05:12:12.329088 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.329063    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a04e4e-6898bff586-28d79"]
Oct 31 05:12:12.671223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:12.671098    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:13.892538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:13.892495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:14.123116 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:14.123077    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" path="/var/lib/kubelet/pods/5acf8b09-b2d2-4a36-a1e0-d438a334ad9a/volumes"
Oct 31 05:12:14.148555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:14.148475    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:14.531212 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:14.531180    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:14.804612 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:14.804523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:15.016707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:15.016659    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:15.229497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:15.229407    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:15.741907 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:15.741849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:15.969289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:15.969252    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:16.202912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:16.202878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:16.623193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:16.623147    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:16.899591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:16.899493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:18.520661 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:18.520621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:19.845101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:19.845053    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:19.932953 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:19.932917    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:12:19.933232 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:19.933200    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" containerName="maas-api" containerID="cri-o://ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987" gracePeriod=30
Oct 31 05:12:20.070231 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.070179    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.176:8443/health\": dial tcp 10.134.0.176:8443: connect: connection refused"
Oct 31 05:12:20.097251 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.097146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:20.231257 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.230801    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:12:20.289382 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.289335    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\") pod \"9add392e-bf73-47f3-83e5-878729b7b21a\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") "
Oct 31 05:12:20.289561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.289458    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\") pod \"9add392e-bf73-47f3-83e5-878729b7b21a\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") "
Oct 31 05:12:20.289614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.289582    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\" (UniqueName: \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\") pod \"9add392e-bf73-47f3-83e5-878729b7b21a\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") "
Oct 31 05:12:20.289667 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.289641    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\") pod \"9add392e-bf73-47f3-83e5-878729b7b21a\" (UID: \"9add392e-bf73-47f3-83e5-878729b7b21a\") "
Oct 31 05:12:20.289818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.289789    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca" pod "9add392e-bf73-47f3-83e5-878729b7b21a" (UID: "9add392e-bf73-47f3-83e5-878729b7b21a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:12:20.290174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.290134    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9add392e-bf73-47f3-83e5-878729b7b21a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:20.296536 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.296503    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs" pod "9add392e-bf73-47f3-83e5-878729b7b21a" (UID: "9add392e-bf73-47f3-83e5-878729b7b21a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:20.297609 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.297579    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls" pod "9add392e-bf73-47f3-83e5-878729b7b21a" (UID: "9add392e-bf73-47f3-83e5-878729b7b21a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:20.299295 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.299265    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll" pod "9add392e-bf73-47f3-83e5-878729b7b21a" (UID: "9add392e-bf73-47f3-83e5-878729b7b21a"). InnerVolumeSpecName "kube-api-access-8vhll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:12:20.310128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.310093    6931 generic.go:356] "Generic (PLEG): container finished" podID="9add392e-bf73-47f3-83e5-878729b7b21a" containerID="ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987" exitCode=0
Oct 31 05:12:20.310510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.310150    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"
Oct 31 05:12:20.310510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.310202    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" event={"ID":"9add392e-bf73-47f3-83e5-878729b7b21a","Type":"ContainerDied","Data":"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"}
Oct 31 05:12:20.310510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.310280    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8" event={"ID":"9add392e-bf73-47f3-83e5-878729b7b21a","Type":"ContainerDied","Data":"f90e23f3ee3f74cb038e7c0941e8c7b880e371786d60b6b85787e07ea9741833"}
Oct 31 05:12:20.310510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.310315    6931 scope.go:122] "RemoveContainer" containerID="ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"
Oct 31 05:12:20.324091 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.324063    6931 scope.go:122] "RemoveContainer" containerID="ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"
Oct 31 05:12:20.324436 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:12:20.324411    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987\": container with ID starting with ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987 not found: ID does not exist" containerID="ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"
Oct 31 05:12:20.324516 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.324451    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987"} err="failed to get container status \"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987\": rpc error: code = NotFound desc = could not find container \"ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987\": container with ID starting with ae7ec7b9d63feecc483be2a3078cfef3ad83bd9c16f2b3602e49a6f022f42987 not found: ID does not exist"
Oct 31 05:12:20.361687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.361608    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:12:20.368394 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.368363    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51cc4ce6-d5d84dcd4-qd6t8"]
Oct 31 05:12:20.391685 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.391644    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:20.391685 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.391682    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vhll\" (UniqueName: \"kubernetes.io/projected/9add392e-bf73-47f3-83e5-878729b7b21a-kube-api-access-8vhll\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:20.391685 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.391691    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9add392e-bf73-47f3-83e5-878729b7b21a-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:20.920622 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:20.920585    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:22.122650 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:22.122613    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" path="/var/lib/kubelet/pods/9add392e-bf73-47f3-83e5-878729b7b21a/volumes"
Oct 31 05:12:22.816499 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:22.816455    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:23.017948 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:23.017913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:23.245699 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:23.245609    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:23.563996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:23.563956    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:23.773027 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:23.772988    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:24.877514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:24.877472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:26.807928 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:26.807890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:28.097186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:28.097136    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:28.317468 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:28.317431    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:28.694986 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:28.694935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:29.001200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:29.001092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:29.434201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:29.431706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:29.646847 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:29.646804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:30.128580 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.128545    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:30.205786 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.205751    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 05:12:30.205994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.205958    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" podUID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" containerName="maas-api" containerID="cri-o://259881fc2a1c1e3e51e3820b86b29110816095b6c73196b3e22bc3c050196960" gracePeriod=30
Oct 31 05:12:30.384943 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.384864    6931 generic.go:356] "Generic (PLEG): container finished" podID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" containerID="259881fc2a1c1e3e51e3820b86b29110816095b6c73196b3e22bc3c050196960" exitCode=0
Oct 31 05:12:30.385087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.385000    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" event={"ID":"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5","Type":"ContainerDied","Data":"259881fc2a1c1e3e51e3820b86b29110816095b6c73196b3e22bc3c050196960"}
Oct 31 05:12:30.448211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.446622    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:30.497704 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.497674    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 05:12:30.626754 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.626713    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\") pod \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") "
Oct 31 05:12:30.626967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.626783    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\") pod \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") "
Oct 31 05:12:30.626967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.626843    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\") pod \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") "
Oct 31 05:12:30.626967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.626911    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\" (UniqueName: \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\") pod \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\" (UID: \"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5\") "
Oct 31 05:12:30.627312 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.627282    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca" pod "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" (UID: "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:12:30.627590 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.627565    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:30.629296 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.629269    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls" pod "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" (UID: "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:30.629540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.629513    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs" pod "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" (UID: "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:30.629700 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.629679    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2" pod "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" (UID: "0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5"). InnerVolumeSpecName "kube-api-access-jmgx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:12:30.728891 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.728844    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:30.728891 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.728880    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:30.729124 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:30.728941    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgx2\" (UniqueName: \"kubernetes.io/projected/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5-kube-api-access-jmgx2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:31.234669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.234635    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:31.392812 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.392780    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"
Oct 31 05:12:31.393011 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.392816    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk" event={"ID":"0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5","Type":"ContainerDied","Data":"19a8441f132aa416180b3a01b066acadfe65a9f7053bb78c2a6e5b7e26246b9d"}
Oct 31 05:12:31.393011 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.392883    6931 scope.go:122] "RemoveContainer" containerID="259881fc2a1c1e3e51e3820b86b29110816095b6c73196b3e22bc3c050196960"
Oct 31 05:12:31.447063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.447029    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 05:12:31.468734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.468679    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-77a153-8c49c4889-tprlk"]
Oct 31 05:12:31.470514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.470481    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:12:31.471650 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471622    6931 cpu_manager.go:414] "containerMap: removing container" podUID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" containerName="maas-api"
Oct 31 05:12:31.471650 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471649    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471674    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471681    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471704    6931 cpu_manager.go:414] "containerMap: removing container" podUID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" containerName="payload-processing"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471712    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" containerName="payload-processing"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471741    6931 cpu_manager.go:414] "containerMap: removing container" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471749    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471792    6931 cpu_manager.go:414] "containerMap: removing container" podUID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471798    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471817    6931 cpu_manager.go:414] "containerMap: removing container" podUID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" containerName="maas-api"
Oct 31 05:12:31.471821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471824    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" containerName="maas-api"
Oct 31 05:12:31.472352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471843    6931 cpu_manager.go:414] "containerMap: removing container" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing"
Oct 31 05:12:31.472352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471850    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing"
Oct 31 05:12:31.472352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471866    6931 cpu_manager.go:414] "containerMap: removing container" podUID="555dc1e4-6853-45a1-90a1-556242c7563c" containerName="payload-pre-processing"
Oct 31 05:12:31.472352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.471874    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555dc1e4-6853-45a1-90a1-556242c7563c" containerName="payload-pre-processing"
Oct 31 05:12:31.472602 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472582    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acf8b09-b2d2-4a36-a1e0-d438a334ad9a" containerName="maas-api"
Oct 31 05:12:31.472660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472610    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9add392e-bf73-47f3-83e5-878729b7b21a" containerName="maas-api"
Oct 31 05:12:31.472660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472625    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfa9411-eaaa-4d6f-b9e1-f2e1b1dfa52c" containerName="payload-processing"
Oct 31 05:12:31.472660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472638    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e33b65-8e7f-4ce6-8199-8c60671a84e4" containerName="payload-pre-processing"
Oct 31 05:12:31.472660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472654    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ec8d57-febf-4ed0-8fe0-6e8b2a48cac0" containerName="maas-api"
Oct 31 05:12:31.472846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472661    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" containerName="maas-api"
Oct 31 05:12:31.472846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472672    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="82010fd1-72b3-46f4-9ec8-f9909ade03b8" containerName="payload-processing"
Oct 31 05:12:31.472846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.472677    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="555dc1e4-6853-45a1-90a1-556242c7563c" containerName="payload-pre-processing"
Oct 31 05:12:31.478763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.478736    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.482981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.482958    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:12:31.493788 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.493758    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"rate-test-model-338f2e-kserve-self-signed-certs\""
Oct 31 05:12:31.493902 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.493870    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"openshift-service-ca.crt\""
Oct 31 05:12:31.493964 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.493950    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"kube-root-ca.crt\""
Oct 31 05:12:31.494051 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.494004    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-338f2e\"/\"default-dockercfg-7vg4l\""
Oct 31 05:12:31.638970 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.638929    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.639169 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.639013    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.639169 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.639082    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.639253 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.639205    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.639320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.639265    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znv8c\" (UniqueName: \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.639401 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.639382    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.648464 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.648433    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:31.741072 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.740969    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741069    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741110    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741147    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znv8c\" (UniqueName: \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741248    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741386    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.741795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.741768    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.742415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.742389    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.742654 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.742624    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.743919 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.743899    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.744385 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.744365    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.764772 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.764742    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znv8c\" (UniqueName: \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\") pod \"rate-test-model-338f2e-kserve-5766b45865-cgwnd\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") " pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:31.795719 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:31.795679    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:32.100305 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.100275    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:12:32.103148 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:12:32.103119    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f640ad6_1426_42c4_8ed3_d5c6ef753214.slice/crio-4c505b3e8aa9ea9950390cdaec47518cda8b5968d22162c7b55ff0f2d9a59798 WatchSource:0}: Error finding container 4c505b3e8aa9ea9950390cdaec47518cda8b5968d22162c7b55ff0f2d9a59798: Status 404 returned error can't find the container with id 4c505b3e8aa9ea9950390cdaec47518cda8b5968d22162c7b55ff0f2d9a59798
Oct 31 05:12:32.124106 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.124072    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5" path="/var/lib/kubelet/pods/0f53ce4e-cd08-4d45-9f68-6cf6b027e4e5/volumes"
Oct 31 05:12:32.402112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.402020    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" event={"ID":"6f640ad6-1426-42c4-8ed3-d5c6ef753214","Type":"ContainerStarted","Data":"2f50f1644dbdb060c919a99c63b5b86cf53a4bb4a12c8431a43d82cb47aa7b7b"}
Oct 31 05:12:32.402112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.402067    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" event={"ID":"6f640ad6-1426-42c4-8ed3-d5c6ef753214","Type":"ContainerStarted","Data":"4c505b3e8aa9ea9950390cdaec47518cda8b5968d22162c7b55ff0f2d9a59798"}
Oct 31 05:12:32.444347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.444281    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" podStartSLOduration=1.4442638859999999 podStartE2EDuration="1.444263886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:12:32.442037638 +0000 UTC m=+4048.475037337" watchObservedRunningTime="2026-10-31 05:12:32.444263886 +0000 UTC m=+4048.477263568"
Oct 31 05:12:32.795994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:32.795953    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:33.037026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:33.036989    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:34.811103 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:34.811056    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:35.715717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:35.715678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:35.928374 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:35.928326    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:36.524602 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:36.524568    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:36.887723 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:36.887682    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:37.127586 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:37.127544    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:37.669477 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:37.669430    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:38.877918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:38.877879    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:40.646742 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:40.646700    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:40.761017 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:12:40.760974    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:12:41.796774 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:41.796740    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:41.811021 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:41.810993    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:42.477629 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:42.477595    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:12:42.510637 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:42.510597    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:43.345823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:43.345779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:43.553877 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:43.553845    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:44.077030 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.076990    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 05:12:44.077308 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.077281    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" podUID="6af0032f-6e1f-44c3-946a-cf96ef798034" containerName="maas-api" containerID="cri-o://7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff" gracePeriod=30
Oct 31 05:12:44.255854 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.255816    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:44.370697 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.370667    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 05:12:44.443180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.439209    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\") pod \"6af0032f-6e1f-44c3-946a-cf96ef798034\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") "
Oct 31 05:12:44.443180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.439278    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\" (UniqueName: \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\") pod \"6af0032f-6e1f-44c3-946a-cf96ef798034\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") "
Oct 31 05:12:44.443180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.439405    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\") pod \"6af0032f-6e1f-44c3-946a-cf96ef798034\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") "
Oct 31 05:12:44.443180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.439481    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") pod \"6af0032f-6e1f-44c3-946a-cf96ef798034\" (UID: \"6af0032f-6e1f-44c3-946a-cf96ef798034\") "
Oct 31 05:12:44.443180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.440901    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca" pod "6af0032f-6e1f-44c3-946a-cf96ef798034" (UID: "6af0032f-6e1f-44c3-946a-cf96ef798034"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:12:44.445727 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.445675    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp" pod "6af0032f-6e1f-44c3-946a-cf96ef798034" (UID: "6af0032f-6e1f-44c3-946a-cf96ef798034"). InnerVolumeSpecName "kube-api-access-hxgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:12:44.452436 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.452364    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls" pod "6af0032f-6e1f-44c3-946a-cf96ef798034" (UID: "6af0032f-6e1f-44c3-946a-cf96ef798034"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:44.453051 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.453016    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs" pod "6af0032f-6e1f-44c3-946a-cf96ef798034" (UID: "6af0032f-6e1f-44c3-946a-cf96ef798034"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:44.487729 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.487682    6931 generic.go:356] "Generic (PLEG): container finished" podID="6af0032f-6e1f-44c3-946a-cf96ef798034" containerID="7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff" exitCode=0
Oct 31 05:12:44.487942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.487815    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" event={"ID":"6af0032f-6e1f-44c3-946a-cf96ef798034","Type":"ContainerDied","Data":"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"}
Oct 31 05:12:44.487942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.487865    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw" event={"ID":"6af0032f-6e1f-44c3-946a-cf96ef798034","Type":"ContainerDied","Data":"3fca809168bd6b4db6be7c3f8a353ea319fbfb142d45e7914bcddc4d2b915f49"}
Oct 31 05:12:44.487942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.487894    6931 scope.go:122] "RemoveContainer" containerID="7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"
Oct 31 05:12:44.488131 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.488056    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"
Oct 31 05:12:44.504490 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.504231    6931 scope.go:122] "RemoveContainer" containerID="7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"
Oct 31 05:12:44.504870 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:12:44.504629    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff\": container with ID starting with 7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff not found: ID does not exist" containerID="7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"
Oct 31 05:12:44.504870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.504672    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff"} err="failed to get container status \"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff\": rpc error: code = NotFound desc = could not find container \"7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff\": container with ID starting with 7344e66a086b1c7d950ea402c050bc2a049bac120a7c559250ca7a16488e16ff not found: ID does not exist"
Oct 31 05:12:44.540760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.540723    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:44.540760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.540757    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af0032f-6e1f-44c3-946a-cf96ef798034-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:44.540760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.540768    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxgnp\" (UniqueName: \"kubernetes.io/projected/6af0032f-6e1f-44c3-946a-cf96ef798034-kube-api-access-hxgnp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:44.541055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.540777    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af0032f-6e1f-44c3-946a-cf96ef798034-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:44.543802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.543778    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 05:12:44.549302 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.549274    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d6ac82-79b868bcb-b4sjw"]
Oct 31 05:12:44.851200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:44.851140    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:45.098507 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:45.098464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:46.122634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:46.122604    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af0032f-6e1f-44c3-946a-cf96ef798034" path="/var/lib/kubelet/pods/6af0032f-6e1f-44c3-946a-cf96ef798034/volumes"
Oct 31 05:12:46.888367 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:46.888325    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:48.605387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:48.605340    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:50.448510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:50.448476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:50.969063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:50.969023    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:51.190416 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:51.190357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:51.775925 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:51.775893    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:52.205784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:52.205749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:52.421948 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:52.421891    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:52.677422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:52.677383    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:54.210388 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:54.210332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:56.009844 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:56.009806    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:57.427017 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.426977    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 05:12:57.427582 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.427244    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" podUID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" containerName="maas-api" containerID="cri-o://679c940849ca2c4182ff726a11339ed53f6cde1ab153a911aed339eda59ed16a" gracePeriod=30
Oct 31 05:12:57.577886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.577850    6931 generic.go:356] "Generic (PLEG): container finished" podID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" containerID="679c940849ca2c4182ff726a11339ed53f6cde1ab153a911aed339eda59ed16a" exitCode=0
Oct 31 05:12:57.578086 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.578065    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" event={"ID":"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9","Type":"ContainerDied","Data":"679c940849ca2c4182ff726a11339ed53f6cde1ab153a911aed339eda59ed16a"}
Oct 31 05:12:57.628759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.628630    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:57.633169 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.633122    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 05:12:57.638491 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.636912    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 05:12:57.725338 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.725313    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 05:12:57.805058 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805020    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\") pod \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") "
Oct 31 05:12:57.805284 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805089    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\") pod \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") "
Oct 31 05:12:57.805284 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805205    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\") pod \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") "
Oct 31 05:12:57.805408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805325    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\" (UniqueName: \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\") pod \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\" (UID: \"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9\") "
Oct 31 05:12:57.805563 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805536    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca" pod "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" (UID: "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:12:57.805842 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.805816    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:57.807901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.807865    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls" pod "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" (UID: "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:57.808028 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.807947    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs" pod "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" (UID: "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:12:57.808355 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.808316    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl" pod "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" (UID: "784baa2a-ac6d-4b86-8f03-d85bb7ca50b9"). InnerVolumeSpecName "kube-api-access-kgvcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:12:57.876330 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.876289    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:57.907175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.907126    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:57.907175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.907170    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:57.907175 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:57.907183    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvcl\" (UniqueName: \"kubernetes.io/projected/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9-kube-api-access-kgvcl\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:12:58.397267 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.397179    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:58.586200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.586139    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"
Oct 31 05:12:58.586681 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.586138    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c" event={"ID":"784baa2a-ac6d-4b86-8f03-d85bb7ca50b9","Type":"ContainerDied","Data":"ab781f69ee6eff45c51107d2593a40905a376e208f891bcf4daadd9bb4e375ac"}
Oct 31 05:12:58.586681 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.586297    6931 scope.go:122] "RemoveContainer" containerID="679c940849ca2c4182ff726a11339ed53f6cde1ab153a911aed339eda59ed16a"
Oct 31 05:12:58.601583 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.601546    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:12:58.638002 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.637961    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 05:12:58.650278 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:12:58.650177    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-031745-5f55f965fc-p6h6c"]
Oct 31 05:13:00.123920 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:00.123880    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" path="/var/lib/kubelet/pods/784baa2a-ac6d-4b86-8f03-d85bb7ca50b9/volumes"
Oct 31 05:13:00.222422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:00.222387    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:02.147108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:02.147061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:03.528150 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:03.528102    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:03.947805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:03.947772    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:04.156687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:04.156653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:04.702863 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:04.702827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:04.949415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:04.949374    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 05:13:04.949904 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:04.949858    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy" containerID="cri-o://835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7" gracePeriod=30
Oct 31 05:13:05.091387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:05.091332    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:05.546384 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:05.546335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:06.246527 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:06.246491    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:08.216408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:08.216368    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:09.987433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:09.987398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:10.254088 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.254056    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 05:13:10.274306 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.274250    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:10.364618 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364579    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.364795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364648    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.364795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364694    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.364795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364730    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.364997 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364803    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.364997 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.364957    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.365108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365005    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.365108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365038    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:10.365108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365055    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.365320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365120    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\") pod \"f95d4201-a147-411b-b427-db5a2f1f7a9e\" (UID: \"f95d4201-a147-411b-b427-db5a2f1f7a9e\") "
Oct 31 05:13:10.365320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365120    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:13:10.365320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365136    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:10.365320 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365258    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:10.365508 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365351    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:10.365706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365684    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95d4201-a147-411b-b427-db5a2f1f7a9e-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.365789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365710    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.365789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365727    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.365789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365741    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.365789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.365753    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.367172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.367139    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:10.367289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.367183    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "kube-api-access-nb8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:10.367289 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.367216    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:10.367556 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.367531    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo" pod "f95d4201-a147-411b-b427-db5a2f1f7a9e" (UID: "f95d4201-a147-411b-b427-db5a2f1f7a9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:13:10.466635 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.466601    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.466635 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.466634    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb8fv\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-kube-api-access-nb8fv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.466880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.466646    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.466880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.466655    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95d4201-a147-411b-b427-db5a2f1f7a9e-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:10.497599 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.497451    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:10.664991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.664956    6931 generic.go:356] "Generic (PLEG): container finished" podID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerID="835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7" exitCode=0
Oct 31 05:13:10.665199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.665067    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" event={"ID":"f95d4201-a147-411b-b427-db5a2f1f7a9e","Type":"ContainerDied","Data":"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"}
Oct 31 05:13:10.665199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.665133    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq" event={"ID":"f95d4201-a147-411b-b427-db5a2f1f7a9e","Type":"ContainerDied","Data":"1377d7b391bd50cfc86a5792aaafac03d1caafc7360ba050c88929e444e0caf0"}
Oct 31 05:13:10.665199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.665195    6931 scope.go:122] "RemoveContainer" containerID="835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"
Oct 31 05:13:10.665376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.665348    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"
Oct 31 05:13:10.678329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.678305    6931 scope.go:122] "RemoveContainer" containerID="835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"
Oct 31 05:13:10.678648 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:10.678628    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7\": container with ID starting with 835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7 not found: ID does not exist" containerID="835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"
Oct 31 05:13:10.678723 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.678659    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7"} err="failed to get container status \"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7\": rpc error: code = NotFound desc = could not find container \"835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7\": container with ID starting with 835b61f9149c7139a30cfcfbfb45c1f9b76d80e5ce4fb4699dc3f3832388b0d7 not found: ID does not exist"
Oct 31 05:13:10.709148 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.709115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:10.717288 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.717256    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 05:13:10.729006 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:10.728973    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-031745-openshift-default-7f667bc645-nf7gq"]
Oct 31 05:13:10.760579 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:10.760497    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:13:11.891811 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:11.891768    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:12.123193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:12.123138    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" path="/var/lib/kubelet/pods/f95d4201-a147-411b-b427-db5a2f1f7a9e/volumes"
Oct 31 05:13:12.664414 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:12.664367    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main" containerID="cri-o://81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924" gracePeriod=45
Oct 31 05:13:12.674334 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:12.674118    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerName="main" containerID="cri-o://276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb" gracePeriod=45
Oct 31 05:13:12.969519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:12.969490    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 05:13:12.973261 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:12.973226    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 05:13:13.098661 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098624    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.098838 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098691    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.098838 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098716    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.098838 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098759    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.099008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098842    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\" (UniqueName: \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.099008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098871    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.099008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098896    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.099008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.098999    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099015    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099024    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099026    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099218    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099078    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.099275 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099256    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.099613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099322    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\" (UniqueName: \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\") pod \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\" (UID: \"8f9403b8-e1b8-4f52-ace8-0bf1714353df\") "
Oct 31 05:13:13.099613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099432    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\") pod \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\" (UID: \"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12\") "
Oct 31 05:13:13.099613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099473    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.099929 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099889    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.100025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099922    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.100025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099946    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.100025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099960    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.100025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099972    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.100025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.099985    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.101674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.101645    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.101784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.101756    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:13.101866 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.101807    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:13.102007 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.101985    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "kube-api-access-9dgnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:13.102134 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.102070    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt" pod "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" (UID: "6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12"). InnerVolumeSpecName "kube-api-access-kskkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:13.102541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.102513    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm" pod "8f9403b8-e1b8-4f52-ace8-0bf1714353df" (UID: "8f9403b8-e1b8-4f52-ace8-0bf1714353df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.201963    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.201995    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskkt\" (UniqueName: \"kubernetes.io/projected/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-kube-api-access-kskkt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.202016    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.202030    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.202044    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9403b8-e1b8-4f52-ace8-0bf1714353df-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.202055    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9403b8-e1b8-4f52-ace8-0bf1714353df-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.202413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.202068    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgnn\" (UniqueName: \"kubernetes.io/projected/8f9403b8-e1b8-4f52-ace8-0bf1714353df-kube-api-access-9dgnn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:13.687507 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:13.687805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687536    6931 generic.go:356] "Generic (PLEG): container finished" podID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerID="81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924" exitCode=0
Oct 31 05:13:13.687805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687670    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"
Oct 31 05:13:13.687805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687756    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" event={"ID":"8f9403b8-e1b8-4f52-ace8-0bf1714353df","Type":"ContainerDied","Data":"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"}
Oct 31 05:13:13.687805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687797    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br" event={"ID":"8f9403b8-e1b8-4f52-ace8-0bf1714353df","Type":"ContainerDied","Data":"cf5fcacf3f13caca3277d7a2063dd9e83f0ad926248371ea5b310881bcd96311"}
Oct 31 05:13:13.687960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.687817    6931 scope.go:122] "RemoveContainer" containerID="81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"
Oct 31 05:13:13.691577 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.691554    6931 generic.go:356] "Generic (PLEG): container finished" podID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerID="276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb" exitCode=0
Oct 31 05:13:13.691674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.691595    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"
Oct 31 05:13:13.691674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.691602    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" event={"ID":"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12","Type":"ContainerDied","Data":"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"}
Oct 31 05:13:13.691674 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.691633    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq" event={"ID":"6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12","Type":"ContainerDied","Data":"8d9adcea5b90a00d72565b806ec95a22de904d55b1540c8e65728bc88e4b4b89"}
Oct 31 05:13:13.703018 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.702918    6931 scope.go:122] "RemoveContainer" containerID="81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"
Oct 31 05:13:13.704993 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:13.703630    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924\": container with ID starting with 81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924 not found: ID does not exist" containerID="81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"
Oct 31 05:13:13.704993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.703674    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924"} err="failed to get container status \"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924\": rpc error: code = NotFound desc = could not find container \"81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924\": container with ID starting with 81b0fb9619b40e949c7a1a6492c9ebc53b890135b334db3ca3a888d222ecf924 not found: ID does not exist"
Oct 31 05:13:13.704993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.703698    6931 scope.go:122] "RemoveContainer" containerID="276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"
Oct 31 05:13:13.715536 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.715510    6931 scope.go:122] "RemoveContainer" containerID="276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"
Oct 31 05:13:13.715909 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:13.715881    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb\": container with ID starting with 276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb not found: ID does not exist" containerID="276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"
Oct 31 05:13:13.716033 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.715920    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb"} err="failed to get container status \"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb\": rpc error: code = NotFound desc = could not find container \"276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb\": container with ID starting with 276eb4ff1c71d299c069b9af23ec87af59d47e446704ac6edf222e8945f38acb not found: ID does not exist"
Oct 31 05:13:13.755463 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.755419    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 05:13:13.758957 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.758931    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-w1-439989-kserve-79c5d77d68-bn2br"]
Oct 31 05:13:13.791669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.791635    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 05:13:13.795836 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.795807    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-439989/distinct-2-w1-439989-kserve-7bc4b5857f-8d2cq"]
Oct 31 05:13:13.966867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:13.966774    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:14.122328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:14.122295    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" path="/var/lib/kubelet/pods/6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12/volumes"
Oct 31 05:13:14.122844 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:14.122822    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" path="/var/lib/kubelet/pods/8f9403b8-e1b8-4f52-ace8-0bf1714353df/volumes"
Oct 31 05:13:15.415309 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:15.415276    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:16.402945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:16.402907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:17.170192 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:17.170127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:17.390341 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:17.390305    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:18.959329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:18.959291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:20.727141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:20.727106    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:22.319096 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319060    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:13:22.319695 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319675    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerName="main"
Oct 31 05:13:22.319759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319699    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerName="main"
Oct 31 05:13:22.319759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319716    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6af0032f-6e1f-44c3-946a-cf96ef798034" containerName="maas-api"
Oct 31 05:13:22.319759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319724    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af0032f-6e1f-44c3-946a-cf96ef798034" containerName="maas-api"
Oct 31 05:13:22.319759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319745    6931 cpu_manager.go:414] "containerMap: removing container" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy"
Oct 31 05:13:22.319759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319752    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy"
Oct 31 05:13:22.319918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319780    6931 cpu_manager.go:414] "containerMap: removing container" podUID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" containerName="maas-api"
Oct 31 05:13:22.319918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319788    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" containerName="maas-api"
Oct 31 05:13:22.319918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319808    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main"
Oct 31 05:13:22.319918 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319815    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main"
Oct 31 05:13:22.320043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319927    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9403b8-e1b8-4f52-ace8-0bf1714353df" containerName="main"
Oct 31 05:13:22.320043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319942    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae8f375-9d23-4b41-ae0f-3d4cd4ae4d12" containerName="main"
Oct 31 05:13:22.320043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319947    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af0032f-6e1f-44c3-946a-cf96ef798034" containerName="maas-api"
Oct 31 05:13:22.320043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319954    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95d4201-a147-411b-b427-db5a2f1f7a9e" containerName="istio-proxy"
Oct 31 05:13:22.320043 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.319960    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="784baa2a-ac6d-4b86-8f03-d85bb7ca50b9" containerName="maas-api"
Oct 31 05:13:22.324879 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.324857    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.346441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.346408    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:13:22.352638 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.352410    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-338f2e-openshift-default-dockercfg-lkdhm\""
Oct 31 05:13:22.404775 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.404730    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:22.406915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.406886    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9h5d\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407054 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.406954    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407058    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407105    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407235 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407136    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407290 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407265    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407327 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407301    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407369 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407328    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.407369 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.407346    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.508758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508713    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.508758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508763    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508801    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508827    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508887    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508914    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508949    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.508977    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509089    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9h5d\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509250    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509527 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509505    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509598 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509536    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509639 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509615    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.509811 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.509788    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.511284 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.511265    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.511711 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.511683    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.533753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.533715    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.533753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.533738    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9h5d\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\") pod \"e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") " pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.629529 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.629447    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:22.638329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.638290    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:22.954511 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.954291    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:13:22.957425 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:13:22.957396    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb18b2d_5cdf_4f8d_9d9f_a8785b488ba3.slice/crio-92d01fb2285e8cd93ed5c641780186c683bcc7ca23118412f3af3e280a20f20f WatchSource:0}: Error finding container 92d01fb2285e8cd93ed5c641780186c683bcc7ca23118412f3af3e280a20f20f: Status 404 returned error can't find the container with id 92d01fb2285e8cd93ed5c641780186c683bcc7ca23118412f3af3e280a20f20f
Oct 31 05:13:22.959909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.959872    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:22.959993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.959942    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:22.959993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:22.959979    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:23.372469 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:23.372430    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:23.763353 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:23.763265    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" event={"ID":"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3","Type":"ContainerStarted","Data":"ddea88d368b21425b5db4ae0e51eb2bbdbac3e938b2dcd583f2f46f8e99d7861"}
Oct 31 05:13:23.763353 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:23.763349    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" event={"ID":"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3","Type":"ContainerStarted","Data":"92d01fb2285e8cd93ed5c641780186c683bcc7ca23118412f3af3e280a20f20f"}
Oct 31 05:13:23.822582 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:23.822493    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podStartSLOduration=1.822455143 podStartE2EDuration="1.822455143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:23.81941643 +0000 UTC m=+4099.852416206" watchObservedRunningTime="2026-10-31 05:13:23.822455143 +0000 UTC m=+4099.855454822"
Oct 31 05:13:24.424285 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:24.424246    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:24.647181 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:24.644216    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:24.647181 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:24.644417    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 05:13:24.647181 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:24.644483    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 05:13:24.655200 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:24.654821    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:25.647602 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:25.647561    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 05:13:25.648244 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:25.648210    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 05:13:26.406402 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:26.406366    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:26.651348 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:26.651299    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 05:13:26.651881 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:26.651363    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 05:13:27.647180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:27.646615    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 05:13:27.647180 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:27.646700    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 05:13:28.306088 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:28.306045    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:28.640687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:28.640588    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 31 05:13:28.640958 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:28.640927    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 31 05:13:29.189457 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:29.189422    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:13:29.376646 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:29.376606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:29.644473 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:29.644433    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:29.644840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:29.644817    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:29.645656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:29.645635    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:13:30.114992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:30.114946    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:30.318773 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:30.318728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:31.489903 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.489862    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 05:13:31.490456 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.490116    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" podUID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" containerName="maas-api" containerID="cri-o://a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8" gracePeriod=30
Oct 31 05:13:31.662252 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.662203    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:31.785151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.785115    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 05:13:31.815409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.815372    6931 generic.go:356] "Generic (PLEG): container finished" podID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" containerID="a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8" exitCode=0
Oct 31 05:13:31.815579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.815418    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"
Oct 31 05:13:31.815579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.815455    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" event={"ID":"8c3e0aa9-2709-41ea-b013-d27f958fe7db","Type":"ContainerDied","Data":"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"}
Oct 31 05:13:31.815579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.815512    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4" event={"ID":"8c3e0aa9-2709-41ea-b013-d27f958fe7db","Type":"ContainerDied","Data":"c82f94cced1825bd4a7f7ba01bc62bb86f7671af93f76d1f0702caa9e6a2001e"}
Oct 31 05:13:31.815579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.815541    6931 scope.go:122] "RemoveContainer" containerID="a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"
Oct 31 05:13:31.825321 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.825256    6931 scope.go:122] "RemoveContainer" containerID="a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"
Oct 31 05:13:31.825592 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:31.825566    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8\": container with ID starting with a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8 not found: ID does not exist" containerID="a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"
Oct 31 05:13:31.825702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.825598    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8"} err="failed to get container status \"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8\": rpc error: code = NotFound desc = could not find container \"a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8\": container with ID starting with a5a0f3b78b4d1bc468732362543001d4fe7756be270f60fc9594b4a4d0d922b8 not found: ID does not exist"
Oct 31 05:13:31.839023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.838894    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\") pod \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") "
Oct 31 05:13:31.839023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.838957    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\" (UniqueName: \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\") pod \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") "
Oct 31 05:13:31.839023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.838989    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\") pod \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") "
Oct 31 05:13:31.839438 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.839196    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") pod \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\" (UID: \"8c3e0aa9-2709-41ea-b013-d27f958fe7db\") "
Oct 31 05:13:31.840073 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.840044    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca" pod "8c3e0aa9-2709-41ea-b013-d27f958fe7db" (UID: "8c3e0aa9-2709-41ea-b013-d27f958fe7db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:13:31.841778 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.841653    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs" pod "8c3e0aa9-2709-41ea-b013-d27f958fe7db" (UID: "8c3e0aa9-2709-41ea-b013-d27f958fe7db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:31.842207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.842145    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls" pod "8c3e0aa9-2709-41ea-b013-d27f958fe7db" (UID: "8c3e0aa9-2709-41ea-b013-d27f958fe7db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:31.843099 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.843072    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h" pod "8c3e0aa9-2709-41ea-b013-d27f958fe7db" (UID: "8c3e0aa9-2709-41ea-b013-d27f958fe7db"). InnerVolumeSpecName "kube-api-access-s6s8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:31.940815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.940773    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3e0aa9-2709-41ea-b013-d27f958fe7db-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:31.940815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.940809    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6s8h\" (UniqueName: \"kubernetes.io/projected/8c3e0aa9-2709-41ea-b013-d27f958fe7db-kube-api-access-s6s8h\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:31.940815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.940821    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:31.941062 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:31.940832    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3e0aa9-2709-41ea-b013-d27f958fe7db-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:32.164128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:32.164093    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 05:13:32.175442 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:32.175409    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-439989-6f8d444988-58qz4"]
Oct 31 05:13:32.421791 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:32.421690    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:34.122872 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:34.122838    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" path="/var/lib/kubelet/pods/8c3e0aa9-2709-41ea-b013-d27f958fe7db/volumes"
Oct 31 05:13:34.179517 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:34.179476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:34.440960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:34.440849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:34.656552 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:34.656508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:36.212514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:36.212470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:36.769637 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:36.769606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:37.101259 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:37.101220    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:37.995433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:37.995388    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:38.375669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:38.373147    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:13:38.517250 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:38.517217    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:38.993303 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:38.993269    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:40.188237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:40.188194    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:40.691362 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:40.691329    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:40.759763 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:40.759725    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:13:40.911225 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:40.911189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:41.462639 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:41.462606    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:41.865089 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:41.865050    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:42.244123 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:42.244030    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:44.211865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.211820    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerName="main" containerID="cri-o://28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31" gracePeriod=45
Oct 31 05:13:44.252064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.252019    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:44.382064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.381680    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": dial tcp 10.134.0.178:8000: connect: connection refused"
Oct 31 05:13:44.512617 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.512586    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:13:44.537776 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.537721    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:44.596293 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596250    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\" (UniqueName: \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596504 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596410    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\" (UniqueName: \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596504 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596452    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596504    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596550    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596614 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596577    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\") pod \"21c89fd6-7cb1-4216-b0db-3d0811971743\" (UID: \"21c89fd6-7cb1-4216-b0db-3d0811971743\") "
Oct 31 05:13:44.596801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596774    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:44.596865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596795    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:44.596920 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.596899    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:44.597105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.597086    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.597186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.597110    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.597186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.597126    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.598641 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.598619    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:44.598723 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.598645    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "kube-api-access-4qllp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:44.598723 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.598670    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm" pod "21c89fd6-7cb1-4216-b0db-3d0811971743" (UID: "21c89fd6-7cb1-4216-b0db-3d0811971743"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:44.698651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.698618    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c89fd6-7cb1-4216-b0db-3d0811971743-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.698651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.698648    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qllp\" (UniqueName: \"kubernetes.io/projected/21c89fd6-7cb1-4216-b0db-3d0811971743-kube-api-access-4qllp\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.698867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.698661    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c89fd6-7cb1-4216-b0db-3d0811971743-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:44.743975 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.743878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:44.908017 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.907979    6931 generic.go:356] "Generic (PLEG): container finished" podID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerID="28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31" exitCode=0
Oct 31 05:13:44.908497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.908049    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"
Oct 31 05:13:44.908497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.908071    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" event={"ID":"21c89fd6-7cb1-4216-b0db-3d0811971743","Type":"ContainerDied","Data":"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"}
Oct 31 05:13:44.908497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.908111    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl" event={"ID":"21c89fd6-7cb1-4216-b0db-3d0811971743","Type":"ContainerDied","Data":"ebbd8b2fc6addf25057f3fb9bb10712b7eccecebf5f3cdddb2cdfe52f368b9e8"}
Oct 31 05:13:44.908497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.908140    6931 scope.go:122] "RemoveContainer" containerID="28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"
Oct 31 05:13:44.922942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.922918    6931 scope.go:122] "RemoveContainer" containerID="28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"
Oct 31 05:13:44.923521 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:44.923492    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31\": container with ID starting with 28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31 not found: ID does not exist" containerID="28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"
Oct 31 05:13:44.923672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.923647    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31"} err="failed to get container status \"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31\": rpc error: code = NotFound desc = could not find container \"28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31\": container with ID starting with 28b384897d1f89d1ea7e8a27437cbd29eabb9b7f237ca8033b8d67e581b63d31 not found: ID does not exist"
Oct 31 05:13:44.959562 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.959521    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:13:44.971067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:44.971029    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-57dc6d/rate-test-model-57dc6d-kserve-85f6f548fc-zn6tl"]
Oct 31 05:13:45.176384 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:45.176345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:46.123029 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:46.122989    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" path="/var/lib/kubelet/pods/21c89fd6-7cb1-4216-b0db-3d0811971743/volumes"
Oct 31 05:13:46.311363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:46.311326    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:48.117810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:48.117763    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:48.325662 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:48.325618    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:49.414686 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:49.414645    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:49.628714 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:49.628673    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:49.843125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:49.843075    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:50.125069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:50.124975    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:51.590812 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:51.590766    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:51.798579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:51.798540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:52.011827 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:52.011729    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:53.398803 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.398761    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerName="main" containerID="cri-o://72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb" gracePeriod=45
Oct 31 05:13:53.490049 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.489993    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.179:8000/ready\": dial tcp 10.134.0.179:8000: connect: connection refused"
Oct 31 05:13:53.673751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.673719    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:13:53.681591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.681561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:53.714040 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.713852    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714266 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714081    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714266 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714143    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714266 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714205    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714266 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714245    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714279    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\" (UniqueName: \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\") pod \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\" (UID: \"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03\") "
Oct 31 05:13:53.714535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714291    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:53.714535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714400    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:53.714535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714506    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:53.714862 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714839    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.714983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714869    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.714983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.714884    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.717346 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.717314    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "kube-api-access-qtk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:13:53.717440 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.717359    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:13:53.717613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.717590    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs" pod "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" (UID: "9f2cb2c0-5771-4d5d-9a1c-20a978f44b03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:13:53.815717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.815678    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.815717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.815716    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.815933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.815729    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtk9v\" (UniqueName: \"kubernetes.io/projected/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03-kube-api-access-qtk9v\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:13:53.962189 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.962065    6931 generic.go:356] "Generic (PLEG): container finished" podID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerID="72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb" exitCode=0
Oct 31 05:13:53.962189 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.962111    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"
Oct 31 05:13:53.962189 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.962148    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" event={"ID":"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03","Type":"ContainerDied","Data":"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"}
Oct 31 05:13:53.962470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.962202    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr" event={"ID":"9f2cb2c0-5771-4d5d-9a1c-20a978f44b03","Type":"ContainerDied","Data":"e8bbe10f638e0cc4a4de349ac69ce7dd0d208e9ee0ab2564519f90c959add884"}
Oct 31 05:13:53.962470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.962219    6931 scope.go:122] "RemoveContainer" containerID="72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"
Oct 31 05:13:53.973004 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.972975    6931 scope.go:122] "RemoveContainer" containerID="72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"
Oct 31 05:13:53.973314 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:13:53.973296    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb\": container with ID starting with 72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb not found: ID does not exist" containerID="72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"
Oct 31 05:13:53.973393 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:53.973324    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb"} err="failed to get container status \"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb\": rpc error: code = NotFound desc = could not find container \"72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb\": container with ID starting with 72221111eb62cd6579b0fdfcfe14e992a2c598e2c938021aaccccd7af5af0bcb not found: ID does not exist"
Oct 31 05:13:54.013245 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:54.013203    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:13:54.019408 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:54.019375    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9633f0/rate-test-model-9633f0-kserve-7cd47f5969-mrcnr"]
Oct 31 05:13:54.112425 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:54.112392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:54.122945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:54.122915    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" path="/var/lib/kubelet/pods/9f2cb2c0-5771-4d5d-9a1c-20a978f44b03/volumes"
Oct 31 05:13:55.224919 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.224882    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:55.337965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.337914    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:13:55.338689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338663    6931 cpu_manager.go:414] "containerMap: removing container" podUID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" containerName="maas-api"
Oct 31 05:13:55.338783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338692    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" containerName="maas-api"
Oct 31 05:13:55.338783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338715    6931 cpu_manager.go:414] "containerMap: removing container" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerName="main"
Oct 31 05:13:55.338783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338724    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerName="main"
Oct 31 05:13:55.338783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338760    6931 cpu_manager.go:414] "containerMap: removing container" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerName="main"
Oct 31 05:13:55.338783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338769    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerName="main"
Oct 31 05:13:55.339082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338902    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3e0aa9-2709-41ea-b013-d27f958fe7db" containerName="maas-api"
Oct 31 05:13:55.339082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338914    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c89fd6-7cb1-4216-b0db-3d0811971743" containerName="main"
Oct 31 05:13:55.339082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.338924    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2cb2c0-5771-4d5d-9a1c-20a978f44b03" containerName="main"
Oct 31 05:13:55.354140 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.353207    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.375982 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.375947    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:13:55.377908 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.377869    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-09a20a-openshift-default-dockercfg-hf9qb\""
Oct 31 05:13:55.440170 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440122    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4n4q\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440350 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440198    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440350 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440288    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440350 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440332    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440410    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440457    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440525 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440488    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440618    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.440690 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.440676    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542377 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542327    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542590 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542388    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542590 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542449    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4n4q\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542590 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542477    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542605    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542654    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542698    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542740    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.542785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542769    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.543032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542799    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.543032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.542904    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.543195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.543080    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.543195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.543129    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.543475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.543450    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.544936 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.544914    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.545233 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.545209    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.565766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.565733    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.566099 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.566071    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4n4q\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\") pod \"e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") " pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.675060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.675013    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:55.980968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.980859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:55.992828 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.992768    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:13:55.996222 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:13:55.996190    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53752427_7039_440d_89e3_6786b914e552.slice/crio-8ca13b5f8572c76d9d94515fddf9b3474b05bea294fc6341637edbf1d6e86cc9 WatchSource:0}: Error finding container 8ca13b5f8572c76d9d94515fddf9b3474b05bea294fc6341637edbf1d6e86cc9: Status 404 returned error can't find the container with id 8ca13b5f8572c76d9d94515fddf9b3474b05bea294fc6341637edbf1d6e86cc9
Oct 31 05:13:55.998914 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.998875    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:55.999002 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.998952    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:55.999002 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:55.998985    6931 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 05:13:56.196655 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:56.196613    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:56.982335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:56.982295    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" event={"ID":"53752427-7039-440d-89e3-6786b914e552","Type":"ContainerStarted","Data":"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"}
Oct 31 05:13:56.982817 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:56.982363    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" event={"ID":"53752427-7039-440d-89e3-6786b914e552","Type":"ContainerStarted","Data":"8ca13b5f8572c76d9d94515fddf9b3474b05bea294fc6341637edbf1d6e86cc9"}
Oct 31 05:13:57.062392 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:57.062330    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podStartSLOduration=2.062316781 podStartE2EDuration="2.062316781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:13:57.049183369 +0000 UTC m=+4133.082183074" watchObservedRunningTime="2026-10-31 05:13:57.062316781 +0000 UTC m=+4133.095316534"
Oct 31 05:13:57.675728 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:57.675690    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:13:57.677459 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:57.677431    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:13:57.677556 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:57.677486    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:13:57.874986 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:57.874944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:58.676092 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:58.675967    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:13:58.676092 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:58.676026    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:13:59.662140 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:59.662099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:13:59.676433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:59.676394    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:13:59.676882 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:13:59.676451    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:14:00.047366 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:00.047240    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:00.255960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:00.255929    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:00.675682 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:00.675639    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:14:00.675890 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:00.675707    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:14:00.772263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:00.772224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:01.161693 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:01.161589    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:01.676540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:01.676498    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:14:01.676860 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:01.676828    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:14:01.789662 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:01.789617    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:02.003489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:02.003402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:02.675619 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:02.675573    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 05:14:02.675862 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:02.675633    6931 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 05:14:03.681435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:03.681402    6931 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:14:03.681963 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:03.681727    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:14:03.682515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:03.682497    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:14:03.703785 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:03.703744    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:04.182131 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:04.182099    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:04.389904 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:04.389861    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:05.501912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:05.501871    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:06.840114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:06.840073    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 05:14:06.840630 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:06.840406    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" podUID="c5df154a-49af-4992-b83b-059b1c03399a" containerName="maas-api" containerID="cri-o://8f3cc0c23b49df04f8e338c46d39b588a347614844f4c2adc16d01923a548a48" gracePeriod=30
Oct 31 05:14:07.056770 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.056729    6931 generic.go:356] "Generic (PLEG): container finished" podID="c5df154a-49af-4992-b83b-059b1c03399a" containerID="8f3cc0c23b49df04f8e338c46d39b588a347614844f4c2adc16d01923a548a48" exitCode=0
Oct 31 05:14:07.056962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.056940    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" event={"ID":"c5df154a-49af-4992-b83b-059b1c03399a","Type":"ContainerDied","Data":"8f3cc0c23b49df04f8e338c46d39b588a347614844f4c2adc16d01923a548a48"}
Oct 31 05:14:07.089639 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.089592    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:07.122632 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.122602    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 05:14:07.210604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.210563    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\") pod \"c5df154a-49af-4992-b83b-059b1c03399a\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") "
Oct 31 05:14:07.210807 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.210625    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\") pod \"c5df154a-49af-4992-b83b-059b1c03399a\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") "
Oct 31 05:14:07.210807 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.210690    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\") pod \"c5df154a-49af-4992-b83b-059b1c03399a\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") "
Oct 31 05:14:07.210807 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.210710    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\" (UniqueName: \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\") pod \"c5df154a-49af-4992-b83b-059b1c03399a\" (UID: \"c5df154a-49af-4992-b83b-059b1c03399a\") "
Oct 31 05:14:07.212041 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.211110    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca" pod "c5df154a-49af-4992-b83b-059b1c03399a" (UID: "c5df154a-49af-4992-b83b-059b1c03399a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:14:07.212041 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.211553    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5df154a-49af-4992-b83b-059b1c03399a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:07.213397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.213366    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs" pod "c5df154a-49af-4992-b83b-059b1c03399a" (UID: "c5df154a-49af-4992-b83b-059b1c03399a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:07.213830 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.213810    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l" pod "c5df154a-49af-4992-b83b-059b1c03399a" (UID: "c5df154a-49af-4992-b83b-059b1c03399a"). InnerVolumeSpecName "kube-api-access-th49l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:14:07.214210 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.214192    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls" pod "c5df154a-49af-4992-b83b-059b1c03399a" (UID: "c5df154a-49af-4992-b83b-059b1c03399a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:07.313057 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.313021    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:07.313057 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.313054    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5df154a-49af-4992-b83b-059b1c03399a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:07.313057 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.313063    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th49l\" (UniqueName: \"kubernetes.io/projected/c5df154a-49af-4992-b83b-059b1c03399a-kube-api-access-th49l\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:07.803831 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:07.803791    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:08.068281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.068185    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"
Oct 31 05:14:08.068281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.068208    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" event={"ID":"c5df154a-49af-4992-b83b-059b1c03399a","Type":"ContainerDied","Data":"111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2"}
Oct 31 05:14:08.068839 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.068285    6931 scope.go:122] "RemoveContainer" containerID="8f3cc0c23b49df04f8e338c46d39b588a347614844f4c2adc16d01923a548a48"
Oct 31 05:14:08.124826 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.124787    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 05:14:08.131892 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.131843    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh"]
Oct 31 05:14:08.136518 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:14:08.136483    6931 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5df154a_49af_4992_b83b_059b1c03399a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5df154a_49af_4992_b83b_059b1c03399a.slice/crio-111f5eaf8dd62a6a0503c4570502d3a76ddc66a356382a54735154d571cc59f2\": RecentStats: unable to find data in memory cache]"
Oct 31 05:14:08.864011 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:08.863973    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:09.069975 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:09.069936    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:09.792749 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:09.792707    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:10.035203 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:10.035127    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-09a20a-5d56bbf584-hc4mh" podUID="c5df154a-49af-4992-b83b-059b1c03399a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.49:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:14:10.123271 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:10.123192    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5df154a-49af-4992-b83b-059b1c03399a" path="/var/lib/kubelet/pods/c5df154a-49af-4992-b83b-059b1c03399a/volumes"
Oct 31 05:14:10.761142 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:14:10.761103    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:14:11.695069 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:11.695034    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:11.980256 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:11.980139    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:14.238015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:14.237935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:14.451481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:14.451444    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:14.587262 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:14.587219    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:14:14.587799 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:14.587759    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" containerID="cri-o://9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1" gracePeriod=30
Oct 31 05:14:14.721009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:14.720955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:15.545293 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:15.545255    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:15.777077 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:15.777035    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:17.332916 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:17.332878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:18.684432 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:18.684391    6931 patch_prober.go:29] interesting pod/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 05:14:18.684988 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:18.684468    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 05:14:19.134666 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.134622    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:19.862571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.862504    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:14:19.962145 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962101    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962378 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962239    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962378 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962305    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962378 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962332    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962560 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962391    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962560 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962411    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:14:19.962560 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962429    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962560 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962524    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:14:19.962751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962531    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962616    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:14:19.962751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962633    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962677    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\") pod \"53752427-7039-440d-89e3-6786b914e552\" (UID: \"53752427-7039-440d-89e3-6786b914e552\") "
Oct 31 05:14:19.962983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962740    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:14:19.962983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.962833    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:14:19.963141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.963122    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:19.963241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.963146    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53752427-7039-440d-89e3-6786b914e552-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:19.963241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.963173    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:19.963241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.963186    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:19.963241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.963199    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:19.964733 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.964695    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:14:19.964733 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.964727    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "kube-api-access-l4n4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:14:19.964868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.964748    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:14:19.964868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:19.964753    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy" pod "53752427-7039-440d-89e3-6786b914e552" (UID: "53752427-7039-440d-89e3-6786b914e552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:14:20.064229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.064194    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53752427-7039-440d-89e3-6786b914e552-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:20.064229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.064224    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53752427-7039-440d-89e3-6786b914e552-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:20.064229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.064233    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:20.064470 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.064241    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4n4q\" (UniqueName: \"kubernetes.io/projected/53752427-7039-440d-89e3-6786b914e552-kube-api-access-l4n4q\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:20.143689 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.143657    6931 generic.go:356] "Generic (PLEG): container finished" podID="53752427-7039-440d-89e3-6786b914e552" containerID="9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1" exitCode=0
Oct 31 05:14:20.143856 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.143701    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"
Oct 31 05:14:20.143856 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.143730    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" event={"ID":"53752427-7039-440d-89e3-6786b914e552","Type":"ContainerDied","Data":"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"}
Oct 31 05:14:20.143856 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.143770    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj" event={"ID":"53752427-7039-440d-89e3-6786b914e552","Type":"ContainerDied","Data":"8ca13b5f8572c76d9d94515fddf9b3474b05bea294fc6341637edbf1d6e86cc9"}
Oct 31 05:14:20.143856 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.143792    6931 scope.go:122] "RemoveContainer" containerID="9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"
Oct 31 05:14:20.153567 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.153543    6931 scope.go:122] "RemoveContainer" containerID="9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"
Oct 31 05:14:20.153876 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:14:20.153858    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1\": container with ID starting with 9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1 not found: ID does not exist" containerID="9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"
Oct 31 05:14:20.153927 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.153886    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1"} err="failed to get container status \"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1\": rpc error: code = NotFound desc = could not find container \"9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1\": container with ID starting with 9a7100a2129b9df00d07178e088ee01a56ef391b2e12868ca9f1697fa1ab6ed1 not found: ID does not exist"
Oct 31 05:14:20.194047 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.194008    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:14:20.205935 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:20.205901    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-09a20a-openshift-default-55d8647ff-czlrj"]
Oct 31 05:14:21.012790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:21.012749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:22.122572 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:22.122537    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53752427-7039-440d-89e3-6786b914e552" path="/var/lib/kubelet/pods/53752427-7039-440d-89e3-6786b914e552/volumes"
Oct 31 05:14:22.828079 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:22.828036    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:24.617707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:24.617667    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:26.411790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:26.411753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:28.295123 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:28.295089    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:30.088302 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:30.088263    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:31.408459 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.408417    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 05:14:31.408965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.408726    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" podUID="a163d702-ad24-4230-9ee7-f1a42e40ec22" containerName="maas-api" containerID="cri-o://b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02" gracePeriod=30
Oct 31 05:14:31.573793 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.573749    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:31.711331 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.711302    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 05:14:31.807760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.807729    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\") pod \"a163d702-ad24-4230-9ee7-f1a42e40ec22\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") "
Oct 31 05:14:31.807934 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.807813    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\") pod \"a163d702-ad24-4230-9ee7-f1a42e40ec22\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") "
Oct 31 05:14:31.807934 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.807840    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") pod \"a163d702-ad24-4230-9ee7-f1a42e40ec22\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") "
Oct 31 05:14:31.808029 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.808010    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\" (UniqueName: \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\") pod \"a163d702-ad24-4230-9ee7-f1a42e40ec22\" (UID: \"a163d702-ad24-4230-9ee7-f1a42e40ec22\") "
Oct 31 05:14:31.808085 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.808063    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca" pod "a163d702-ad24-4230-9ee7-f1a42e40ec22" (UID: "a163d702-ad24-4230-9ee7-f1a42e40ec22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:14:31.808439 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.808418    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a163d702-ad24-4230-9ee7-f1a42e40ec22-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:31.809999 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.809972    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv" pod "a163d702-ad24-4230-9ee7-f1a42e40ec22" (UID: "a163d702-ad24-4230-9ee7-f1a42e40ec22"). InnerVolumeSpecName "kube-api-access-bmfxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:14:31.810107 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.809996    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs" pod "a163d702-ad24-4230-9ee7-f1a42e40ec22" (UID: "a163d702-ad24-4230-9ee7-f1a42e40ec22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:31.810107 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.810011    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls" pod "a163d702-ad24-4230-9ee7-f1a42e40ec22" (UID: "a163d702-ad24-4230-9ee7-f1a42e40ec22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:31.910299 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.910249    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfxv\" (UniqueName: \"kubernetes.io/projected/a163d702-ad24-4230-9ee7-f1a42e40ec22-kube-api-access-bmfxv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:31.910532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.910355    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:31.910532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:31.910373    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a163d702-ad24-4230-9ee7-f1a42e40ec22-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:32.215715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.215679    6931 generic.go:356] "Generic (PLEG): container finished" podID="a163d702-ad24-4230-9ee7-f1a42e40ec22" containerID="b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02" exitCode=0
Oct 31 05:14:32.215937 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.215727    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"
Oct 31 05:14:32.215937 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.215776    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" event={"ID":"a163d702-ad24-4230-9ee7-f1a42e40ec22","Type":"ContainerDied","Data":"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"}
Oct 31 05:14:32.215937 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.215838    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn" event={"ID":"a163d702-ad24-4230-9ee7-f1a42e40ec22","Type":"ContainerDied","Data":"a095a21d9896ddd55cf78c2712286b77f9a27fedaa6f8264df56fe32a5a92203"}
Oct 31 05:14:32.215937 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.215867    6931 scope.go:122] "RemoveContainer" containerID="b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"
Oct 31 05:14:32.225571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.225552    6931 scope.go:122] "RemoveContainer" containerID="b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"
Oct 31 05:14:32.225880 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:14:32.225857    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02\": container with ID starting with b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02 not found: ID does not exist" containerID="b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"
Oct 31 05:14:32.225941 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.225889    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02"} err="failed to get container status \"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02\": rpc error: code = NotFound desc = could not find container \"b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02\": container with ID starting with b1ffd0a7c9e4bd954e189f315e3223662ed12d7c14ee173d9d9416980b523a02 not found: ID does not exist"
Oct 31 05:14:32.266512 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.266479    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 05:14:32.278362 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.278321    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ded435-67cf854c44-mzwxn"]
Oct 31 05:14:32.345410 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:32.345377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:34.122776 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:34.122737    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a163d702-ad24-4230-9ee7-f1a42e40ec22" path="/var/lib/kubelet/pods/a163d702-ad24-4230-9ee7-f1a42e40ec22/volumes"
Oct 31 05:14:34.196444 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:34.196410    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:36.005763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:36.005722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:37.795257 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:37.795211    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:38.005761 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:38.005725    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:38.691487 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:38.691444    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:39.072880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:39.072846    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:39.717359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:39.717321    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:40.162919 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:40.162880    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:40.759373 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:14:40.759339    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:14:41.522422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:41.522388    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:43.307143 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:43.307092    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:45.285919 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:45.285887    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:46.636402 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:46.636358    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:46.860135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:46.860098    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:47.086172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:47.086127    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:47.299112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:47.299041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:49.104884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:49.104849    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:50.903608 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:50.903562    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:52.640360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:52.640324    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:53.038517 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:53.038473    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:54.370573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:54.370537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:56.147688 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:56.147645    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:57.860010 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:57.859953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:59.191381 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.191344    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 05:14:59.191863 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.191573    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" podUID="38b43381-e151-4f9e-90d7-be115c805ff7" containerName="maas-api" containerID="cri-o://afed65625fc2611d8cb3bdefac5116c019e641b3d710b3b9f1a9665843b12266" gracePeriod=30
Oct 31 05:14:59.359957 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.359866    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:14:59.385357 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.384928    6931 generic.go:356] "Generic (PLEG): container finished" podID="38b43381-e151-4f9e-90d7-be115c805ff7" containerID="afed65625fc2611d8cb3bdefac5116c019e641b3d710b3b9f1a9665843b12266" exitCode=0
Oct 31 05:14:59.385357 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.385125    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" event={"ID":"38b43381-e151-4f9e-90d7-be115c805ff7","Type":"ContainerDied","Data":"afed65625fc2611d8cb3bdefac5116c019e641b3d710b3b9f1a9665843b12266"}
Oct 31 05:14:59.488115 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.488084    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 05:14:59.630936 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.630895    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\") pod \"38b43381-e151-4f9e-90d7-be115c805ff7\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") "
Oct 31 05:14:59.631129 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.631039    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\" (UniqueName: \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\") pod \"38b43381-e151-4f9e-90d7-be115c805ff7\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") "
Oct 31 05:14:59.631129 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.631113    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\") pod \"38b43381-e151-4f9e-90d7-be115c805ff7\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") "
Oct 31 05:14:59.631260 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.631214    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") pod \"38b43381-e151-4f9e-90d7-be115c805ff7\" (UID: \"38b43381-e151-4f9e-90d7-be115c805ff7\") "
Oct 31 05:14:59.631568 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.631535    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca" pod "38b43381-e151-4f9e-90d7-be115c805ff7" (UID: "38b43381-e151-4f9e-90d7-be115c805ff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:14:59.631787 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.631762    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b43381-e151-4f9e-90d7-be115c805ff7-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:59.633232 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.633208    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs" pod "38b43381-e151-4f9e-90d7-be115c805ff7" (UID: "38b43381-e151-4f9e-90d7-be115c805ff7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:59.633363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.633345    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls" pod "38b43381-e151-4f9e-90d7-be115c805ff7" (UID: "38b43381-e151-4f9e-90d7-be115c805ff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:14:59.633474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.633454    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj" pod "38b43381-e151-4f9e-90d7-be115c805ff7" (UID: "38b43381-e151-4f9e-90d7-be115c805ff7"). InnerVolumeSpecName "kube-api-access-fmhfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:14:59.733484 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.733401    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:59.733484 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.733435    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhfj\" (UniqueName: \"kubernetes.io/projected/38b43381-e151-4f9e-90d7-be115c805ff7-kube-api-access-fmhfj\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:14:59.733484 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:14:59.733446    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b43381-e151-4f9e-90d7-be115c805ff7-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:00.020782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.020686    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:00.179207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179136    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:00.179656 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179643    6931 cpu_manager.go:414] "containerMap: removing container" podUID="a163d702-ad24-4230-9ee7-f1a42e40ec22" containerName="maas-api"
Oct 31 05:15:00.179702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179658    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a163d702-ad24-4230-9ee7-f1a42e40ec22" containerName="maas-api"
Oct 31 05:15:00.179702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179681    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c5df154a-49af-4992-b83b-059b1c03399a" containerName="maas-api"
Oct 31 05:15:00.179702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179687    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5df154a-49af-4992-b83b-059b1c03399a" containerName="maas-api"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179715    6931 cpu_manager.go:414] "containerMap: removing container" podUID="38b43381-e151-4f9e-90d7-be115c805ff7" containerName="maas-api"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179721    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b43381-e151-4f9e-90d7-be115c805ff7" containerName="maas-api"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179728    6931 cpu_manager.go:414] "containerMap: removing container" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179733    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179815    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b43381-e151-4f9e-90d7-be115c805ff7" containerName="maas-api"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179824    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="53752427-7039-440d-89e3-6786b914e552" containerName="istio-proxy"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179833    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="a163d702-ad24-4230-9ee7-f1a42e40ec22" containerName="maas-api"
Oct 31 05:15:00.179846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.179841    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5df154a-49af-4992-b83b-059b1c03399a" containerName="maas-api"
Oct 31 05:15:00.184441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.184039    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:00.184441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.184311    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.189148 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.189123    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.204254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.204222    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:00.209571 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.209541    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:00.209944 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.209923    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.216993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.216960    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:00.233397 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.233360    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:00.276509 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.276430    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzb8r\""
Oct 31 05:15:00.340439 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340400    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnqr\" (UniqueName: \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.340646 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340478    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92gq\" (UniqueName: \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.340646 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340569    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmrf\" (UniqueName: \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.340646 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340628    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.340784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340714    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.340825 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.340808    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.391998 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.391967    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"
Oct 31 05:15:00.391998 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.391986    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52" event={"ID":"38b43381-e151-4f9e-90d7-be115c805ff7","Type":"ContainerDied","Data":"56283308acc583a28ed00959e08774221dcb2a14ff7be15f4685878628fff396"}
Oct 31 05:15:00.392242 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.392051    6931 scope.go:122] "RemoveContainer" containerID="afed65625fc2611d8cb3bdefac5116c019e641b3d710b3b9f1a9665843b12266"
Oct 31 05:15:00.441782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.441744    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnqr\" (UniqueName: \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.441983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.441832    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92gq\" (UniqueName: \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.441983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.441869    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmrf\" (UniqueName: \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.441983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.441935    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.441983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.441961    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.442258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.442055    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.442671 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.442648    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.442784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.442757    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.442850 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.442827    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.445807 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.445780    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 05:15:00.461120 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.461094    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-57dc6d-865c88d97f-s2l52"]
Oct 31 05:15:00.467038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.467010    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmrf\" (UniqueName: \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\") pod \"maas-api-key-cleanup-29890395-b22hl\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.467204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.467063    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92gq\" (UniqueName: \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\") pod \"maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.467204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.467072    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnqr\" (UniqueName: \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\") pod \"maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.508489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.508452    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:00.523729 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.523699    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:00.538121 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.538089    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:00.836233 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.836201    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:00.839183 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:15:00.839141    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9cd816_5815_4568_9bca_fdceb1a56b69.slice/crio-1c934eaf549c2bff87cae5cdb50677b52a8caedcad2558c87949fa65907d9c24 WatchSource:0}: Error finding container 1c934eaf549c2bff87cae5cdb50677b52a8caedcad2558c87949fa65907d9c24: Status 404 returned error can't find the container with id 1c934eaf549c2bff87cae5cdb50677b52a8caedcad2558c87949fa65907d9c24
Oct 31 05:15:00.885347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.885305    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:00.889563 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:15:00.889534    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46f9fbf_5f62_4a23_8774_65abd361dcc2.slice/crio-78bef1e3454248497c582f0731c5836410da841275b9080341acd7e83983d3d7 WatchSource:0}: Error finding container 78bef1e3454248497c582f0731c5836410da841275b9080341acd7e83983d3d7: Status 404 returned error can't find the container with id 78bef1e3454248497c582f0731c5836410da841275b9080341acd7e83983d3d7
Oct 31 05:15:00.907866 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:00.907837    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:00.911599 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:15:00.911569    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3194872f_4856_4b5c_a6c5_a38e58a0c2d7.slice/crio-7a151854fbf28aa736a2073a03008754560e6c4cf2b16553eb6f3d7e57067865 WatchSource:0}: Error finding container 7a151854fbf28aa736a2073a03008754560e6c4cf2b16553eb6f3d7e57067865: Status 404 returned error can't find the container with id 7a151854fbf28aa736a2073a03008754560e6c4cf2b16553eb6f3d7e57067865
Oct 31 05:15:01.399503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.399403    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerStarted","Data":"65c8f086495dd61a8b49327255913a8cc6fba5192fe5bf7cd99b2a6bd40101bf"}
Oct 31 05:15:01.399503 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.399456    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerStarted","Data":"1c934eaf549c2bff87cae5cdb50677b52a8caedcad2558c87949fa65907d9c24"}
Oct 31 05:15:01.401042 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.401017    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerStarted","Data":"a4ce84162a39e4b1aa4dee96f0e1cd1a44ffd66583ed3ae0d48f244ac8fdb006"}
Oct 31 05:15:01.401125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.401067    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerStarted","Data":"78bef1e3454248497c582f0731c5836410da841275b9080341acd7e83983d3d7"}
Oct 31 05:15:01.402639 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.402612    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerStarted","Data":"c8be5ef3d3ccfc016707ce5d0f80f5cff25ee634d1b8d03b7cd521b4a984eebc"}
Oct 31 05:15:01.402777 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.402651    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerStarted","Data":"7a151854fbf28aa736a2073a03008754560e6c4cf2b16553eb6f3d7e57067865"}
Oct 31 05:15:01.439181 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.439103    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" podStartSLOduration=1.43908793 podStartE2EDuration="1.43908793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.437244474 +0000 UTC m=+4197.470244155" watchObservedRunningTime="2026-10-31 05:15:01.43908793 +0000 UTC m=+4197.472087609"
Oct 31 05:15:01.476879 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.476827    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" podStartSLOduration=1.476811448 podStartE2EDuration="1.476811448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.474720896 +0000 UTC m=+4197.507720574" watchObservedRunningTime="2026-10-31 05:15:01.476811448 +0000 UTC m=+4197.509811126"
Oct 31 05:15:01.512802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.512743    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" podStartSLOduration=1.512726537 podStartE2EDuration="1.512726537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:01.512480787 +0000 UTC m=+4197.545480485" watchObservedRunningTime="2026-10-31 05:15:01.512726537 +0000 UTC m=+4197.545726215"
Oct 31 05:15:01.953325 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:01.953275    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:02.122668 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.122625    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b43381-e151-4f9e-90d7-be115c805ff7" path="/var/lib/kubelet/pods/38b43381-e151-4f9e-90d7-be115c805ff7/volumes"
Oct 31 05:15:02.411672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.411582    6931 generic.go:356] "Generic (PLEG): container finished" podID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerID="65c8f086495dd61a8b49327255913a8cc6fba5192fe5bf7cd99b2a6bd40101bf" exitCode=60
Oct 31 05:15:02.412125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.411664    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerDied","Data":"65c8f086495dd61a8b49327255913a8cc6fba5192fe5bf7cd99b2a6bd40101bf"}
Oct 31 05:15:02.412125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.411935    6931 scope.go:122] "RemoveContainer" containerID="65c8f086495dd61a8b49327255913a8cc6fba5192fe5bf7cd99b2a6bd40101bf"
Oct 31 05:15:02.413314 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.413292    6931 generic.go:356] "Generic (PLEG): container finished" podID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerID="a4ce84162a39e4b1aa4dee96f0e1cd1a44ffd66583ed3ae0d48f244ac8fdb006" exitCode=60
Oct 31 05:15:02.413410 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.413367    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerDied","Data":"a4ce84162a39e4b1aa4dee96f0e1cd1a44ffd66583ed3ae0d48f244ac8fdb006"}
Oct 31 05:15:02.413573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.413559    6931 scope.go:122] "RemoveContainer" containerID="a4ce84162a39e4b1aa4dee96f0e1cd1a44ffd66583ed3ae0d48f244ac8fdb006"
Oct 31 05:15:02.415306 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.415282    6931 generic.go:356] "Generic (PLEG): container finished" podID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerID="c8be5ef3d3ccfc016707ce5d0f80f5cff25ee634d1b8d03b7cd521b4a984eebc" exitCode=60
Oct 31 05:15:02.415406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.415334    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerDied","Data":"c8be5ef3d3ccfc016707ce5d0f80f5cff25ee634d1b8d03b7cd521b4a984eebc"}
Oct 31 05:15:02.415523 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:02.415511    6931 scope.go:122] "RemoveContainer" containerID="c8be5ef3d3ccfc016707ce5d0f80f5cff25ee634d1b8d03b7cd521b4a984eebc"
Oct 31 05:15:03.424006 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:03.423965    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerStarted","Data":"d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"}
Oct 31 05:15:03.426122 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:03.426093    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerStarted","Data":"cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"}
Oct 31 05:15:03.429735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:03.428430    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerStarted","Data":"75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"}
Oct 31 05:15:03.737698 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:03.737608    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:04.305395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.305358    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:04.437538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.437502    6931 generic.go:356] "Generic (PLEG): container finished" podID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerID="d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370" exitCode=60
Oct 31 05:15:04.438009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.437586    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerDied","Data":"d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"}
Oct 31 05:15:04.438009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.437643    6931 scope.go:122] "RemoveContainer" containerID="c8be5ef3d3ccfc016707ce5d0f80f5cff25ee634d1b8d03b7cd521b4a984eebc"
Oct 31 05:15:04.438009 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.437930    6931 scope.go:122] "RemoveContainer" containerID="d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"
Oct 31 05:15:04.438333 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:04.438312    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh_odh-ai-gateway-infra(3194872f-4856-4b5c-a6c5-a38e58a0c2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7"
Oct 31 05:15:04.439541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.439521    6931 generic.go:356] "Generic (PLEG): container finished" podID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerID="cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0" exitCode=60
Oct 31 05:15:04.439643 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.439589    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerDied","Data":"cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"}
Oct 31 05:15:04.439794 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.439779    6931 scope.go:122] "RemoveContainer" containerID="cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"
Oct 31 05:15:04.440034 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:04.440016    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9_odh-ai-gateway-infra(2c9cd816-5815-4568-9bca-fdceb1a56b69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69"
Oct 31 05:15:04.441334 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.441312    6931 generic.go:356] "Generic (PLEG): container finished" podID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerID="75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c" exitCode=60
Oct 31 05:15:04.441488 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.441380    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerDied","Data":"75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"}
Oct 31 05:15:04.441543 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.441533    6931 scope.go:122] "RemoveContainer" containerID="75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"
Oct 31 05:15:04.441738 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:04.441719    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890395-b22hl_odh-ai-gateway-infra(c46f9fbf-5f62-4a23-8774-65abd361dcc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2"
Oct 31 05:15:04.449427 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.449407    6931 scope.go:122] "RemoveContainer" containerID="65c8f086495dd61a8b49327255913a8cc6fba5192fe5bf7cd99b2a6bd40101bf"
Oct 31 05:15:04.462207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:04.462186    6931 scope.go:122] "RemoveContainer" containerID="a4ce84162a39e4b1aa4dee96f0e1cd1a44ffd66583ed3ae0d48f244ac8fdb006"
Oct 31 05:15:05.451637 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:05.451606    6931 scope.go:122] "RemoveContainer" containerID="d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"
Oct 31 05:15:05.452044 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:05.451986    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh_odh-ai-gateway-infra(3194872f-4856-4b5c-a6c5-a38e58a0c2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7"
Oct 31 05:15:05.454323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:05.454295    6931 scope.go:122] "RemoveContainer" containerID="cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"
Oct 31 05:15:05.454718 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:05.454686    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9_odh-ai-gateway-infra(2c9cd816-5815-4568-9bca-fdceb1a56b69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69"
Oct 31 05:15:05.456406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:05.456383    6931 scope.go:122] "RemoveContainer" containerID="75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"
Oct 31 05:15:05.456690 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:05.456668    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890395-b22hl_odh-ai-gateway-infra(c46f9fbf-5f62-4a23-8774-65abd361dcc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2"
Oct 31 05:15:05.490212 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:05.490168    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:07.274118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:07.274079    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:07.554055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:07.554000    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:07.768147 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:07.768096    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:08.341696 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:08.341663    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:08.716815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:08.716712    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:09.236453 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:09.236412    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:09.811555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:09.811513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:10.759189 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:10.759126    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:15:10.987844 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:10.987804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:12.829127 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:12.829088    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:14.537494 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:14.537456    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:15.634041 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:15.634006    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:15.848591 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:15.848551    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:16.115457 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:16.115421    6931 scope.go:122] "RemoveContainer" containerID="cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"
Oct 31 05:15:16.521475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:16.521441    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerStarted","Data":"2c00b9572a2ab3a82df1c863671bfdea7d3068e379ec0621b2639a1447516e52"}
Oct 31 05:15:16.760593 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:16.760506    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:16.972334 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:16.972296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:17.115986 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.115951    6931 scope.go:122] "RemoveContainer" containerID="d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"
Oct 31 05:15:17.116228 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.116042    6931 scope.go:122] "RemoveContainer" containerID="75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"
Oct 31 05:15:17.531300 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.531267    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerStarted","Data":"d0c911144c15cd79a4b61a3624689f1aa5516c37fae26f3590898ff4c3e3a491"}
Oct 31 05:15:17.538031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.537999    6931 generic.go:356] "Generic (PLEG): container finished" podID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerID="2c00b9572a2ab3a82df1c863671bfdea7d3068e379ec0621b2639a1447516e52" exitCode=60
Oct 31 05:15:17.538223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.538058    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerDied","Data":"2c00b9572a2ab3a82df1c863671bfdea7d3068e379ec0621b2639a1447516e52"}
Oct 31 05:15:17.538223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.538105    6931 scope.go:122] "RemoveContainer" containerID="cabbef4dd50294135958810871f6f8c8fc7362881bcc1c911cf4342c1c6376c0"
Oct 31 05:15:17.538504 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.538487    6931 scope.go:122] "RemoveContainer" containerID="2c00b9572a2ab3a82df1c863671bfdea7d3068e379ec0621b2639a1447516e52"
Oct 31 05:15:17.538730 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:17.538703    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9_odh-ai-gateway-infra(2c9cd816-5815-4568-9bca-fdceb1a56b69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69"
Oct 31 05:15:17.570036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.570003    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:17.585917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.585867    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-c7f796"]
Oct 31 05:15:17.591103 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.591076    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:17.608020 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.605399    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-prswc\""
Oct 31 05:15:17.611842 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.611802    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-c7f796"]
Oct 31 05:15:17.637802 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.637760    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mr2\" (UniqueName: \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\") pod \"test-curl-7096-c7f796\" (UID: \"189dcf95-97d2-478f-9cd4-d25bbfa1dec1\") " pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:17.739434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.739354    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mr2\" (UniqueName: \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\") pod \"test-curl-7096-c7f796\" (UID: \"189dcf95-97d2-478f-9cd4-d25bbfa1dec1\") " pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:17.762053 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.762018    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mr2\" (UniqueName: \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\") pod \"test-curl-7096-c7f796\" (UID: \"189dcf95-97d2-478f-9cd4-d25bbfa1dec1\") " pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:17.921942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:17.921906    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:18.275840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.275795    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-c7f796"]
Oct 31 05:15:18.279365 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:15:18.279339    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189dcf95_97d2_478f_9cd4_d25bbfa1dec1.slice/crio-d71b9457b2fc46557dd00d21b0f90d388ba277a1e4233e718335311d2ede9363 WatchSource:0}: Error finding container d71b9457b2fc46557dd00d21b0f90d388ba277a1e4233e718335311d2ede9363: Status 404 returned error can't find the container with id d71b9457b2fc46557dd00d21b0f90d388ba277a1e4233e718335311d2ede9363
Oct 31 05:15:18.546126 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.546089    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-c7f796" event={"ID":"189dcf95-97d2-478f-9cd4-d25bbfa1dec1","Type":"ContainerStarted","Data":"8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8"}
Oct 31 05:15:18.546126 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.546135    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-c7f796" event={"ID":"189dcf95-97d2-478f-9cd4-d25bbfa1dec1","Type":"ContainerStarted","Data":"d71b9457b2fc46557dd00d21b0f90d388ba277a1e4233e718335311d2ede9363"}
Oct 31 05:15:18.551723 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.551694    6931 generic.go:356] "Generic (PLEG): container finished" podID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerID="5e8cf02d5274ed52539fdb05e3bae0a05a71c7e91b110480af973bbbf751ec09" exitCode=60
Oct 31 05:15:18.551864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.551766    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerDied","Data":"5e8cf02d5274ed52539fdb05e3bae0a05a71c7e91b110480af973bbbf751ec09"}
Oct 31 05:15:18.551864 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.551830    6931 scope.go:122] "RemoveContainer" containerID="75992df924b6ee3f8a94b2c03feef83076168c8142c089c3125f98475c7ee54c"
Oct 31 05:15:18.552179 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.552147    6931 scope.go:122] "RemoveContainer" containerID="5e8cf02d5274ed52539fdb05e3bae0a05a71c7e91b110480af973bbbf751ec09"
Oct 31 05:15:18.552454 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:18.552426    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890395-b22hl_odh-ai-gateway-infra(c46f9fbf-5f62-4a23-8774-65abd361dcc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2"
Oct 31 05:15:18.553743 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.553720    6931 generic.go:356] "Generic (PLEG): container finished" podID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerID="d0c911144c15cd79a4b61a3624689f1aa5516c37fae26f3590898ff4c3e3a491" exitCode=60
Oct 31 05:15:18.553835 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.553758    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerDied","Data":"d0c911144c15cd79a4b61a3624689f1aa5516c37fae26f3590898ff4c3e3a491"}
Oct 31 05:15:18.554090 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.554074    6931 scope.go:122] "RemoveContainer" containerID="d0c911144c15cd79a4b61a3624689f1aa5516c37fae26f3590898ff4c3e3a491"
Oct 31 05:15:18.554347 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:18.554320    6931 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh_odh-ai-gateway-infra(3194872f-4856-4b5c-a6c5-a38e58a0c2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7"
Oct 31 05:15:18.565424 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.565401    6931 scope.go:122] "RemoveContainer" containerID="d4e029ac0146c9d3f3694345663c8e46fdcface32f4b103e78a6ea641133d370"
Oct 31 05:15:18.584931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.584875    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-c7f796" podStartSLOduration=1.584859614 podStartE2EDuration="1.584859614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:18.58288299 +0000 UTC m=+4214.615882669" watchObservedRunningTime="2026-10-31 05:15:18.584859614 +0000 UTC m=+4214.617859287"
Oct 31 05:15:18.589561 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.589533    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:18.690472 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.690442    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:18.692948 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.692913    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:18.752197 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.752148    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\") pod \"2c9cd816-5815-4568-9bca-fdceb1a56b69\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") "
Oct 31 05:15:18.752406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.752391    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\" (UniqueName: \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\") pod \"2c9cd816-5815-4568-9bca-fdceb1a56b69\" (UID: \"2c9cd816-5815-4568-9bca-fdceb1a56b69\") "
Oct 31 05:15:18.752545 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.752521    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca" pod "2c9cd816-5815-4568-9bca-fdceb1a56b69" (UID: "2c9cd816-5815-4568-9bca-fdceb1a56b69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:15:18.752740 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.752724    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cd816-5815-4568-9bca-fdceb1a56b69-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:18.754546 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.754524    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr" pod "2c9cd816-5815-4568-9bca-fdceb1a56b69" (UID: "2c9cd816-5815-4568-9bca-fdceb1a56b69"). InnerVolumeSpecName "kube-api-access-ngnqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:18.853734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:18.853652    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnqr\" (UniqueName: \"kubernetes.io/projected/2c9cd816-5815-4568-9bca-fdceb1a56b69-kube-api-access-ngnqr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:19.449622 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.449534    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-c7f796"]
Oct 31 05:15:19.455707 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.455667    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-c7f796"]
Oct 31 05:15:19.561031 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.560999    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"
Oct 31 05:15:19.561285 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.560996    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9" event={"ID":"2c9cd816-5815-4568-9bca-fdceb1a56b69","Type":"ContainerDied","Data":"1c934eaf549c2bff87cae5cdb50677b52a8caedcad2558c87949fa65907d9c24"}
Oct 31 05:15:19.561285 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.561109    6931 scope.go:122] "RemoveContainer" containerID="2c00b9572a2ab3a82df1c863671bfdea7d3068e379ec0621b2639a1447516e52"
Oct 31 05:15:19.610898 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.610863    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:19.620235 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.620202    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-227c93-29890395-psss9"]
Oct 31 05:15:19.622303 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622280    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-024b07"]
Oct 31 05:15:19.622853 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622835    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.622895 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622869    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.622946 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622896    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.622946 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622904    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.622946 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622924    6931 cpu_manager.go:414] "containerMap: removing container" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" containerName="test-curl-7096-c7f796"
Oct 31 05:15:19.622946 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.622932    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" containerName="test-curl-7096-c7f796"
Oct 31 05:15:19.623138 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.623089    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.623138 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.623105    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:15:19.623138 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.623116    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" containerName="test-curl-7096-c7f796"
Oct 31 05:15:19.630430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.630404    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:19.643283 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.643252    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-024b07"]
Oct 31 05:15:19.663665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.663625    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdf9d\" (UniqueName: \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\") pod \"test-curl-7096-024b07\" (UID: \"785982dc-fdb6-4553-b782-ffef892542b8\") " pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:19.672355 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.672324    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:19.732017 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.731989    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:19.765150 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.765112    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\") pod \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") "
Oct 31 05:15:19.765365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.765202    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\" (UniqueName: \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\") pod \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\" (UID: \"3194872f-4856-4b5c-a6c5-a38e58a0c2d7\") "
Oct 31 05:15:19.765446 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.765432    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdf9d\" (UniqueName: \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\") pod \"test-curl-7096-024b07\" (UID: \"785982dc-fdb6-4553-b782-ffef892542b8\") " pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:19.765559 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.765538    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca" pod "3194872f-4856-4b5c-a6c5-a38e58a0c2d7" (UID: "3194872f-4856-4b5c-a6c5-a38e58a0c2d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:15:19.765692 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.765675    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:19.767847 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.767822    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq" pod "3194872f-4856-4b5c-a6c5-a38e58a0c2d7" (UID: "3194872f-4856-4b5c-a6c5-a38e58a0c2d7"). InnerVolumeSpecName "kube-api-access-t92gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:19.788223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.788186    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdf9d\" (UniqueName: \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\") pod \"test-curl-7096-024b07\" (UID: \"785982dc-fdb6-4553-b782-ffef892542b8\") " pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:19.820147 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.820118    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:19.867016 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.866951    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\") pod \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") "
Oct 31 05:15:19.867548 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.867108    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\" (UniqueName: \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\") pod \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\" (UID: \"c46f9fbf-5f62-4a23-8774-65abd361dcc2\") "
Oct 31 05:15:19.867548 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.867386    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca" pod "c46f9fbf-5f62-4a23-8774-65abd361dcc2" (UID: "c46f9fbf-5f62-4a23-8774-65abd361dcc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:15:19.867548 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.867509    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92gq\" (UniqueName: \"kubernetes.io/projected/3194872f-4856-4b5c-a6c5-a38e58a0c2d7-kube-api-access-t92gq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:19.867548 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.867524    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46f9fbf-5f62-4a23-8774-65abd361dcc2-maas-api-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:19.869423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.869400    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf" pod "c46f9fbf-5f62-4a23-8774-65abd361dcc2" (UID: "c46f9fbf-5f62-4a23-8774-65abd361dcc2"). InnerVolumeSpecName "kube-api-access-5qmrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:19.946824 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.946783    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:19.968737 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:19.968704    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmrf\" (UniqueName: \"kubernetes.io/projected/c46f9fbf-5f62-4a23-8774-65abd361dcc2-kube-api-access-5qmrf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:20.122482 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.122444    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" path="/var/lib/kubelet/pods/2c9cd816-5815-4568-9bca-fdceb1a56b69/volumes"
Oct 31 05:15:20.250475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.250397    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-024b07"]
Oct 31 05:15:20.253818 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:15:20.253789    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod785982dc_fdb6_4553_b782_ffef892542b8.slice/crio-ffc56faed702d970a37c3d67a8995fde4d1c7188d520046426daeacc7cc0d378 WatchSource:0}: Error finding container ffc56faed702d970a37c3d67a8995fde4d1c7188d520046426daeacc7cc0d378: Status 404 returned error can't find the container with id ffc56faed702d970a37c3d67a8995fde4d1c7188d520046426daeacc7cc0d378
Oct 31 05:15:20.495432 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.495392    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:20.573413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.573374    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl" event={"ID":"c46f9fbf-5f62-4a23-8774-65abd361dcc2","Type":"ContainerDied","Data":"78bef1e3454248497c582f0731c5836410da841275b9080341acd7e83983d3d7"}
Oct 31 05:15:20.573598 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.573471    6931 scope.go:122] "RemoveContainer" containerID="5e8cf02d5274ed52539fdb05e3bae0a05a71c7e91b110480af973bbbf751ec09"
Oct 31 05:15:20.573598 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.573393    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"
Oct 31 05:15:20.575094 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.575030    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-024b07" event={"ID":"785982dc-fdb6-4553-b782-ffef892542b8","Type":"ContainerStarted","Data":"486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a"}
Oct 31 05:15:20.575094 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.575069    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-024b07" event={"ID":"785982dc-fdb6-4553-b782-ffef892542b8","Type":"ContainerStarted","Data":"ffc56faed702d970a37c3d67a8995fde4d1c7188d520046426daeacc7cc0d378"}
Oct 31 05:15:20.576520 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.576494    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh" event={"ID":"3194872f-4856-4b5c-a6c5-a38e58a0c2d7","Type":"ContainerDied","Data":"7a151854fbf28aa736a2073a03008754560e6c4cf2b16553eb6f3d7e57067865"}
Oct 31 05:15:20.576642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.576570    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-c7f796" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" containerName="test-curl-7096-c7f796" containerID="cri-o://8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8" gracePeriod=2
Oct 31 05:15:20.576642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.576610    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"
Oct 31 05:15:20.584256 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.584229    6931 scope.go:122] "RemoveContainer" containerID="d0c911144c15cd79a4b61a3624689f1aa5516c37fae26f3590898ff4c3e3a491"
Oct 31 05:15:20.622953 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.622918    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:20.626801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.626774    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890395-b22hl"]
Oct 31 05:15:20.658953 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.658918    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:20.662917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.662892    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4c2387-29890395-pv9fh"]
Oct 31 05:15:20.706407 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:20.706339    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-024b07" podStartSLOduration=1.706322927 podStartE2EDuration="1.706322927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:15:20.704714291 +0000 UTC m=+4216.737713995" watchObservedRunningTime="2026-10-31 05:15:20.706322927 +0000 UTC m=+4216.739322630"
Oct 31 05:15:21.577603 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:21.577529    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-024b07"]
Oct 31 05:15:21.581729 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:21.581674    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-024b07"]
Oct 31 05:15:22.122947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.122914    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" path="/var/lib/kubelet/pods/3194872f-4856-4b5c-a6c5-a38e58a0c2d7/volumes"
Oct 31 05:15:22.123404 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.123387    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" path="/var/lib/kubelet/pods/c46f9fbf-5f62-4a23-8774-65abd361dcc2/volumes"
Oct 31 05:15:22.453668 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.453579    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:22.592826 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.592747    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-024b07" podUID="785982dc-fdb6-4553-b782-ffef892542b8" containerName="test-curl-7096-024b07" containerID="cri-o://486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a" gracePeriod=2
Oct 31 05:15:22.768647 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.768622    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:22.902488 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.902448    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\" (UniqueName: \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\") pod \"189dcf95-97d2-478f-9cd4-d25bbfa1dec1\" (UID: \"189dcf95-97d2-478f-9cd4-d25bbfa1dec1\") "
Oct 31 05:15:22.904764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:22.904736    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2" pod "189dcf95-97d2-478f-9cd4-d25bbfa1dec1" (UID: "189dcf95-97d2-478f-9cd4-d25bbfa1dec1"). InnerVolumeSpecName "kube-api-access-w7mr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:23.003720 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.003641    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mr2\" (UniqueName: \"kubernetes.io/projected/189dcf95-97d2-478f-9cd4-d25bbfa1dec1-kube-api-access-w7mr2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:23.495352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.495304    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:23.599254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.599218    6931 generic.go:356] "Generic (PLEG): container finished" podID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" containerID="8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8" exitCode=137
Oct 31 05:15:23.599709 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.599262    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:23.599709 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.599277    6931 scope.go:122] "RemoveContainer" containerID="8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8"
Oct 31 05:15:23.609213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.609188    6931 scope.go:122] "RemoveContainer" containerID="8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8"
Oct 31 05:15:23.609526 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:23.609503    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8\": container with ID starting with 8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8 not found: ID does not exist" containerID="8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8"
Oct 31 05:15:23.609611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:23.609536    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8"} err="failed to get container status \"8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8\": rpc error: code = NotFound desc = could not find container \"8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8\": container with ID starting with 8d8cbd06a0cd7e33e9022588ca40e6f1b545d81f8c023fdec603360f9cec86c8 not found: ID does not exist"
Oct 31 05:15:23.621822 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:23.621787    6931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-c7f796\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:24.122455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.122366    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" path="/var/lib/kubelet/pods/189dcf95-97d2-478f-9cd4-d25bbfa1dec1/volumes"
Oct 31 05:15:24.130607 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:24.130569    6931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-c7f796\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="189dcf95-97d2-478f-9cd4-d25bbfa1dec1" pod="opendatahub/test-curl-7096-c7f796"
Oct 31 05:15:24.305984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.305938    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:24.769389 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.769355    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:24.822583 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.822554    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\" (UniqueName: \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\") pod \"785982dc-fdb6-4553-b782-ffef892542b8\" (UID: \"785982dc-fdb6-4553-b782-ffef892542b8\") "
Oct 31 05:15:24.824884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.824847    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d" pod "785982dc-fdb6-4553-b782-ffef892542b8" (UID: "785982dc-fdb6-4553-b782-ffef892542b8"). InnerVolumeSpecName "kube-api-access-sdf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:24.924584 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:24.924494    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdf9d\" (UniqueName: \"kubernetes.io/projected/785982dc-fdb6-4553-b782-ffef892542b8-kube-api-access-sdf9d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:25.614863 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:25.614829    6931 generic.go:356] "Generic (PLEG): container finished" podID="785982dc-fdb6-4553-b782-ffef892542b8" containerID="486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a" exitCode=137
Oct 31 05:15:25.615064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:25.614896    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:25.615064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:25.614937    6931 scope.go:122] "RemoveContainer" containerID="486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a"
Oct 31 05:15:25.624058 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:25.624031    6931 scope.go:122] "RemoveContainer" containerID="486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a"
Oct 31 05:15:25.624364 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:25.624344    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a\": container with ID starting with 486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a not found: ID does not exist" containerID="486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a"
Oct 31 05:15:25.624432 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:25.624373    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a"} err="failed to get container status \"486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a\": rpc error: code = NotFound desc = could not find container \"486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a\": container with ID starting with 486d921cb3a58d3fa2447a1faee74dc83524b883eaa2282872a04ab9eb35c31a not found: ID does not exist"
Oct 31 05:15:25.636715 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:25.636684    6931 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-024b07\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="785982dc-fdb6-4553-b782-ffef892542b8" pod="opendatahub/test-curl-7096-024b07"
Oct 31 05:15:26.101558 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:26.101521    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:26.124429 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:26.124395    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785982dc-fdb6-4553-b782-ffef892542b8" path="/var/lib/kubelet/pods/785982dc-fdb6-4553-b782-ffef892542b8/volumes"
Oct 31 05:15:27.818786 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:27.818750    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:29.622978 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:29.622940    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:31.389536 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:31.389495    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:33.284790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:33.284736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:35.200668 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:35.200626    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:36.680460 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:36.680418    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:37.333223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:37.333189    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:38.774927 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:38.774890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:39.049600 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:39.049565    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:39.262970 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:39.262924    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:39.524423 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:39.524328    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:39.964537 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:39.964498    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:40.407679 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:40.407641    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:40.759600 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:15:40.759519    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:15:41.090263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:41.090218    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:42.633038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:42.632996    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:15:42.776941 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:42.776906    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:42.995062 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:42.994970    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:44.894022 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:44.893915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:45.467940 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:45.467905    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:45.784277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:45.784240    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:46.279722 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:46.279632    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:46.802334 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:46.802303    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:48.157613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:48.157577    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:48.599335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:48.599296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:50.311949 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:50.311912    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:51.297989 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:51.297932    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:52.201492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:52.201452    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:52.489981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:52.489890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:52.705814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:52.705775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:53.338747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:53.338709    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:53.550762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:53.550728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:54.418496 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:54.418459    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:55.192710 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:55.192677    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:56.411193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:56.411136    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:57.655694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:57.655625    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" podUID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" containerName="main" containerID="cri-o://2f50f1644dbdb060c919a99c63b5b86cf53a4bb4a12c8431a43d82cb47aa7b7b" gracePeriod=45
Oct 31 05:15:57.809913 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:57.809871    6931 generic.go:356] "Generic (PLEG): container finished" podID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" containerID="2f50f1644dbdb060c919a99c63b5b86cf53a4bb4a12c8431a43d82cb47aa7b7b" exitCode=0
Oct 31 05:15:57.810094 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:57.809932    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" event={"ID":"6f640ad6-1426-42c4-8ed3-d5c6ef753214","Type":"ContainerDied","Data":"2f50f1644dbdb060c919a99c63b5b86cf53a4bb4a12c8431a43d82cb47aa7b7b"}
Oct 31 05:15:57.914332 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:57.914260    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:15:58.022962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.022927    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.022998    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023059    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023140    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\" (UniqueName: \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023210    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023249    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\") pod \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\" (UID: \"6f640ad6-1426-42c4-8ed3-d5c6ef753214\") "
Oct 31 05:15:58.023412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023354    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:15:58.023578 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023489    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:15:58.023640 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023590    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:15:58.023766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023734    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.023766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023746    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.023766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.023756    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.025532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.025510    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:15:58.025752 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.025731    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:15:58.025820 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.025733    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c" pod "6f640ad6-1426-42c4-8ed3-d5c6ef753214" (UID: "6f640ad6-1426-42c4-8ed3-d5c6ef753214"). InnerVolumeSpecName "kube-api-access-znv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:15:58.124871 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.124837    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f640ad6-1426-42c4-8ed3-d5c6ef753214-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.124871 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.124865    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f640ad6-1426-42c4-8ed3-d5c6ef753214-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.124871 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.124877    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znv8c\" (UniqueName: \"kubernetes.io/projected/6f640ad6-1426-42c4-8ed3-d5c6ef753214-kube-api-access-znv8c\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:15:58.263021 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.262915    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:15:58.817201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.817142    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"
Oct 31 05:15:58.817201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.817191    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd" event={"ID":"6f640ad6-1426-42c4-8ed3-d5c6ef753214","Type":"ContainerDied","Data":"4c505b3e8aa9ea9950390cdaec47518cda8b5968d22162c7b55ff0f2d9a59798"}
Oct 31 05:15:58.817721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.817247    6931 scope.go:122] "RemoveContainer" containerID="2f50f1644dbdb060c919a99c63b5b86cf53a4bb4a12c8431a43d82cb47aa7b7b"
Oct 31 05:15:58.869226 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.869189    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:15:58.881167 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:58.881119    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-338f2e/rate-test-model-338f2e-kserve-5766b45865-cgwnd"]
Oct 31 05:15:59.020654 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:15:59.020612    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:00.122618 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:00.122584    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" path="/var/lib/kubelet/pods/6f640ad6-1426-42c4-8ed3-d5c6ef753214/volumes"
Oct 31 05:16:00.146694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:00.146660    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:01.243127 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:01.243085    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:01.900008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:01.899968    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:03.738012 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:03.737971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:04.438322 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:04.438283    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:05.270237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:05.270197    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:16:05.386510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:05.386467    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:05.611329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:05.611288    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:06.196036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:06.195992    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:07.376352 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:07.376305    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:07.616474 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:07.616427    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:16:07.762118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:07.762021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:08.014672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:08.014558    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:08.217853 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:08.217814    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:09.796737 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:09.796699    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:10.759623 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:10.759587    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:16:11.631990 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:11.631947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:13.433524 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:13.433492    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:14.060060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:14.060026    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:14.282243 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:14.282203    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:15.113034 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:15.112997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:15.520123 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:15.520041    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:16.289490 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:16.289447    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:16.505250 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:16.505215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:17.070441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:17.070401    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:17.601917 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:17.601880    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:17.813522 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:17.813468    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:18.310167 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:18.310110    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:19.207968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.207930    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:19.506997 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.506891    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 05:16:19.507226 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.507184    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" podUID="4c31e963-b71e-4edb-a266-0baf09315cd6" containerName="maas-api" containerID="cri-o://07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee" gracePeriod=30
Oct 31 05:16:19.677540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.677508    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:19.805688 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.805661    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 05:16:19.879468 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.879432    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\") pod \"4c31e963-b71e-4edb-a266-0baf09315cd6\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") "
Oct 31 05:16:19.879670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.879477    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\") pod \"4c31e963-b71e-4edb-a266-0baf09315cd6\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") "
Oct 31 05:16:19.879670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.879524    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\" (UniqueName: \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\") pod \"4c31e963-b71e-4edb-a266-0baf09315cd6\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") "
Oct 31 05:16:19.879670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.879645    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\") pod \"4c31e963-b71e-4edb-a266-0baf09315cd6\" (UID: \"4c31e963-b71e-4edb-a266-0baf09315cd6\") "
Oct 31 05:16:19.880113 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.880087    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca" pod "4c31e963-b71e-4edb-a266-0baf09315cd6" (UID: "4c31e963-b71e-4edb-a266-0baf09315cd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:16:19.882286 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.882256    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs" pod "4c31e963-b71e-4edb-a266-0baf09315cd6" (UID: "4c31e963-b71e-4edb-a266-0baf09315cd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:19.882411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.882265    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls" pod "4c31e963-b71e-4edb-a266-0baf09315cd6" (UID: "4c31e963-b71e-4edb-a266-0baf09315cd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:19.882510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.882485    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6" pod "4c31e963-b71e-4edb-a266-0baf09315cd6" (UID: "4c31e963-b71e-4edb-a266-0baf09315cd6"). InnerVolumeSpecName "kube-api-access-p8ht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:16:19.946822 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.946785    6931 generic.go:356] "Generic (PLEG): container finished" podID="4c31e963-b71e-4edb-a266-0baf09315cd6" containerID="07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee" exitCode=0
Oct 31 05:16:19.946962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.946829    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"
Oct 31 05:16:19.946962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.946866    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" event={"ID":"4c31e963-b71e-4edb-a266-0baf09315cd6","Type":"ContainerDied","Data":"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"}
Oct 31 05:16:19.946962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.946918    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62" event={"ID":"4c31e963-b71e-4edb-a266-0baf09315cd6","Type":"ContainerDied","Data":"7cf30e46582837df26889690b429620c36dde3c83c094cd9835f3de04cb9bde4"}
Oct 31 05:16:19.946962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.946939    6931 scope.go:122] "RemoveContainer" containerID="07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"
Oct 31 05:16:19.957054 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.957034    6931 scope.go:122] "RemoveContainer" containerID="07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"
Oct 31 05:16:19.957372 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:19.957351    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee\": container with ID starting with 07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee not found: ID does not exist" containerID="07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"
Oct 31 05:16:19.957453 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.957384    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee"} err="failed to get container status \"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee\": rpc error: code = NotFound desc = could not find container \"07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee\": container with ID starting with 07023cd305a0f6566b0737583f987895656394d7bf1539da1dd725a5b31612ee not found: ID does not exist"
Oct 31 05:16:19.981202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.981140    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c31e963-b71e-4edb-a266-0baf09315cd6-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:19.981202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.981203    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:19.981426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.981217    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c31e963-b71e-4edb-a266-0baf09315cd6-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:19.981426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:19.981229    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ht6\" (UniqueName: \"kubernetes.io/projected/4c31e963-b71e-4edb-a266-0baf09315cd6-kube-api-access-p8ht6\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.001137 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.001100    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 05:16:20.005253 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.004519    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8973e2-7b76888646-prj62"]
Oct 31 05:16:20.122865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.122830    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c31e963-b71e-4edb-a266-0baf09315cd6" path="/var/lib/kubelet/pods/4c31e963-b71e-4edb-a266-0baf09315cd6/volumes"
Oct 31 05:16:20.298949 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.298913    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" podUID="bc0203d3-e103-4d9a-a98c-ff68b4606859" containerName="main" containerID="cri-o://74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea" gracePeriod=45
Oct 31 05:16:20.407622 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.407527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:20.528263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.528239    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:16:20.587037 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587006    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587096    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\" (UniqueName: \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587135    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587272 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587252    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587449 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587329    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587509 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587475    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\") pod \"bc0203d3-e103-4d9a-a98c-ff68b4606859\" (UID: \"bc0203d3-e103-4d9a-a98c-ff68b4606859\") "
Oct 31 05:16:20.587605 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587556    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:20.587673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587620    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:20.587816 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587795    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:20.587951 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587929    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.587994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587961    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.587994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.587978    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.589483 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.589459    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:20.589592 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.589523    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:20.589592 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.589550    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d" pod "bc0203d3-e103-4d9a-a98c-ff68b4606859" (UID: "bc0203d3-e103-4d9a-a98c-ff68b4606859"). InnerVolumeSpecName "kube-api-access-vgf2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:16:20.689280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.689202    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0203d3-e103-4d9a-a98c-ff68b4606859-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.689280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.689232    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgf2d\" (UniqueName: \"kubernetes.io/projected/bc0203d3-e103-4d9a-a98c-ff68b4606859-kube-api-access-vgf2d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.689280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.689241    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0203d3-e103-4d9a-a98c-ff68b4606859-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:20.954644 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.954552    6931 generic.go:356] "Generic (PLEG): container finished" podID="bc0203d3-e103-4d9a-a98c-ff68b4606859" containerID="74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea" exitCode=0
Oct 31 05:16:20.954644 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.954601    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"
Oct 31 05:16:20.954857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.954642    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" event={"ID":"bc0203d3-e103-4d9a-a98c-ff68b4606859","Type":"ContainerDied","Data":"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"}
Oct 31 05:16:20.954857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.954688    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl" event={"ID":"bc0203d3-e103-4d9a-a98c-ff68b4606859","Type":"ContainerDied","Data":"983e4b88a19e4d3d6e4de5c01ba76341245bd02eeac979b096f2849df7f47c09"}
Oct 31 05:16:20.954857 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.954709    6931 scope.go:122] "RemoveContainer" containerID="74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"
Oct 31 05:16:20.967172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.967138    6931 scope.go:122] "RemoveContainer" containerID="74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"
Oct 31 05:16:20.967794 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:20.967773    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea\": container with ID starting with 74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea not found: ID does not exist" containerID="74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"
Oct 31 05:16:20.967874 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:20.967802    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea"} err="failed to get container status \"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea\": rpc error: code = NotFound desc = could not find container \"74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea\": container with ID starting with 74bf0f724ac62fcae701f8f4964d64de7b00d9667188c6e01b5c7fc13ce98eea not found: ID does not exist"
Oct 31 05:16:21.004357 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:21.004327    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:16:21.015868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:21.015831    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-766728/sub-test-model-766728-kserve-6f7d896bfc-qhxcl"]
Oct 31 05:16:21.693221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:21.693188    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:21.982642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:21.982560    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:22.122459 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.122424    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0203d3-e103-4d9a-a98c-ff68b4606859" path="/var/lib/kubelet/pods/bc0203d3-e103-4d9a-a98c-ff68b4606859/volumes"
Oct 31 05:16:22.543736 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.543687    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:22.639801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.639638    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" podUID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" containerName="main" containerID="cri-o://9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177" gracePeriod=45
Oct 31 05:16:22.770026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.769993    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:22.957911 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.957887    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:16:22.968867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.968833    6931 generic.go:356] "Generic (PLEG): container finished" podID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" containerID="9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177" exitCode=0
Oct 31 05:16:22.969026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.968873    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"
Oct 31 05:16:22.969026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.968911    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" event={"ID":"425452e2-5ddf-4958-8eb1-d4fd1ebba711","Type":"ContainerDied","Data":"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"}
Oct 31 05:16:22.969026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.968956    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl" event={"ID":"425452e2-5ddf-4958-8eb1-d4fd1ebba711","Type":"ContainerDied","Data":"9216dce07b339c56d9babfbf4fc62c68acb3b41bc69c99b6efe9d7de2280add1"}
Oct 31 05:16:22.969026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.968974    6931 scope.go:122] "RemoveContainer" containerID="9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"
Oct 31 05:16:22.978280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.978255    6931 scope.go:122] "RemoveContainer" containerID="9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"
Oct 31 05:16:22.978589 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:22.978570    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177\": container with ID starting with 9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177 not found: ID does not exist" containerID="9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"
Oct 31 05:16:22.978661 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.978606    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177"} err="failed to get container status \"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177\": rpc error: code = NotFound desc = could not find container \"9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177\": container with ID starting with 9e1642eda60ea8cf0a9d2b836fa2db6a8684d21d00982bb5a2192e4e8e462177 not found: ID does not exist"
Oct 31 05:16:22.988788 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:22.988763    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:23.124202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.124082    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.124202 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.124146    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\" (UniqueName: \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.124455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.124222    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\" (UniqueName: \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.124789 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.124766    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.125071 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.125042    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:23.125273 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.125242    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.125405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.125367    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\") pod \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\" (UID: \"425452e2-5ddf-4958-8eb1-d4fd1ebba711\") "
Oct 31 05:16:23.125810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.125787    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.126025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.126004    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:23.126223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.126204    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:23.126874 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.126846    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:16:23.126964 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.126946    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "kube-api-access-5d6qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:16:23.137280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.137249    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs" pod "425452e2-5ddf-4958-8eb1-d4fd1ebba711" (UID: "425452e2-5ddf-4958-8eb1-d4fd1ebba711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:23.226762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.226724    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.226762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.226762    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.227039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.226778    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d6qn\" (UniqueName: \"kubernetes.io/projected/425452e2-5ddf-4958-8eb1-d4fd1ebba711-kube-api-access-5d6qn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.227039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.226815    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/425452e2-5ddf-4958-8eb1-d4fd1ebba711-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.227039 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.226830    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/425452e2-5ddf-4958-8eb1-d4fd1ebba711-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:23.289798 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.289757    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:23.317715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.317373    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:16:23.325821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.325781    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-25f56e/sub-test-model-25f56e-kserve-5b4cd89879-2d5gl"]
Oct 31 05:16:23.520330 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.520245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:23.736360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.736319    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:23.947324 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:23.947281    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:24.122268 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:24.122234    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" path="/var/lib/kubelet/pods/425452e2-5ddf-4958-8eb1-d4fd1ebba711/volumes"
Oct 31 05:16:24.656405 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:24.656362    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:24.926296 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:24.926203    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:25.141434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:25.141393    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:25.402753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:25.402716    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:25.630595 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:25.630555    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:26.198432 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:26.198396    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:26.621505 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:26.621465    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:26.837430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:26.837383    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:27.185634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:27.185596    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:28.603555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:28.603509    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:29.914920 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:29.914883    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:30.123347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:30.123306    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:30.367285 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:30.367247    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:31.001575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:31.001538    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:31.728203 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:31.728137    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:31.947323 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:31.947286    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:32.893758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:32.893720    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:33.367340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:33.367302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:33.590387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:33.590349    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:34.147734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:34.147695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:34.536475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:34.536432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:35.081194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:35.081130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:35.382047 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:35.381941    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:36.638786 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:36.638752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:36.846269 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:36.846224    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:38.183930 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.183890    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 05:16:38.184452 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.184138    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" podUID="955d9376-e518-4754-8434-175eb794b910" containerName="maas-api" containerID="cri-o://292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0" gracePeriod=30
Oct 31 05:16:38.359250 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.359205    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:38.458136 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.458066    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 05:16:38.501625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.500560    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\") pod \"955d9376-e518-4754-8434-175eb794b910\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") "
Oct 31 05:16:38.501625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.500666    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\") pod \"955d9376-e518-4754-8434-175eb794b910\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") "
Oct 31 05:16:38.501625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.500755    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\" (UniqueName: \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\") pod \"955d9376-e518-4754-8434-175eb794b910\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") "
Oct 31 05:16:38.501625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.500820    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\") pod \"955d9376-e518-4754-8434-175eb794b910\" (UID: \"955d9376-e518-4754-8434-175eb794b910\") "
Oct 31 05:16:38.501625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.501568    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca" pod "955d9376-e518-4754-8434-175eb794b910" (UID: "955d9376-e518-4754-8434-175eb794b910"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:16:38.503766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.503740    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw" pod "955d9376-e518-4754-8434-175eb794b910" (UID: "955d9376-e518-4754-8434-175eb794b910"). InnerVolumeSpecName "kube-api-access-7mtdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:16:38.503927 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.503775    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs" pod "955d9376-e518-4754-8434-175eb794b910" (UID: "955d9376-e518-4754-8434-175eb794b910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:38.503927 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.503847    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls" pod "955d9376-e518-4754-8434-175eb794b910" (UID: "955d9376-e518-4754-8434-175eb794b910"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:38.602288 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.602256    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:38.602288 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.602284    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/955d9376-e518-4754-8434-175eb794b910-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:38.602288 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.602296    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtdw\" (UniqueName: \"kubernetes.io/projected/955d9376-e518-4754-8434-175eb794b910-kube-api-access-7mtdw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:38.602547 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:38.602306    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/955d9376-e518-4754-8434-175eb794b910-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:39.043312 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.043274    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:39.056137 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.056103    6931 generic.go:356] "Generic (PLEG): container finished" podID="955d9376-e518-4754-8434-175eb794b910" containerID="292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0" exitCode=0
Oct 31 05:16:39.056328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.056174    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"
Oct 31 05:16:39.056328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.056194    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" event={"ID":"955d9376-e518-4754-8434-175eb794b910","Type":"ContainerDied","Data":"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"}
Oct 31 05:16:39.056328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.056269    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph" event={"ID":"955d9376-e518-4754-8434-175eb794b910","Type":"ContainerDied","Data":"88c93e9992c2bdf8ae2c7cf58343f08cfbc13ae4994e6594c650e93b395aa2af"}
Oct 31 05:16:39.056328 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.056291    6931 scope.go:122] "RemoveContainer" containerID="292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"
Oct 31 05:16:39.067037 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.067016    6931 scope.go:122] "RemoveContainer" containerID="292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"
Oct 31 05:16:39.067331 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:39.067313    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0\": container with ID starting with 292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0 not found: ID does not exist" containerID="292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"
Oct 31 05:16:39.067393 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.067340    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0"} err="failed to get container status \"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0\": rpc error: code = NotFound desc = could not find container \"292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0\": container with ID starting with 292a4610cced17c896d320cac2cfed8a4343c073df24564a03cf7ee995a8eff0 not found: ID does not exist"
Oct 31 05:16:39.107684 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.107652    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 05:16:39.119640 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:39.119611    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-227c93-79f96d5b57-dllph"]
Oct 31 05:16:40.121836 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:40.121803    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955d9376-e518-4754-8434-175eb794b910" path="/var/lib/kubelet/pods/955d9376-e518-4754-8434-175eb794b910/volumes"
Oct 31 05:16:40.765932 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:40.765868    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:16:40.811865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:40.811827    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:41.482340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:41.482296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:41.697217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:41.697142    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:41.911143 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:41.911100    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:42.726255 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:42.726216    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:42.949743 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:42.949636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:43.447697 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:43.447654    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:43.843778 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:43.843740    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:44.627376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:44.627337    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:44.899987 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:44.899896    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:46.600178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:46.600120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:47.647054 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:47.647013    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:48.409058 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:48.409025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:50.029726 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:50.029684    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:50.684171 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:50.684119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:50.897898 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:50.897863    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:51.407940 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:51.407907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:52.227801 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.227761    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 05:16:52.228341 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.228303    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" podUID="390db20a-4526-419e-9304-7f67e47cbe83" containerName="maas-api" containerID="cri-o://d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7" gracePeriod=30
Oct 31 05:16:52.379797 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.379709    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:52.502444 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.502418    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 05:16:52.667216 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.667179    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\" (UniqueName: \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\") pod \"390db20a-4526-419e-9304-7f67e47cbe83\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") "
Oct 31 05:16:52.667418 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.667402    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\") pod \"390db20a-4526-419e-9304-7f67e47cbe83\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") "
Oct 31 05:16:52.667509 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.667495    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\") pod \"390db20a-4526-419e-9304-7f67e47cbe83\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") "
Oct 31 05:16:52.667575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.667549    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\") pod \"390db20a-4526-419e-9304-7f67e47cbe83\" (UID: \"390db20a-4526-419e-9304-7f67e47cbe83\") "
Oct 31 05:16:52.667967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.667912    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca" pod "390db20a-4526-419e-9304-7f67e47cbe83" (UID: "390db20a-4526-419e-9304-7f67e47cbe83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:16:52.670144 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.670078    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr" pod "390db20a-4526-419e-9304-7f67e47cbe83" (UID: "390db20a-4526-419e-9304-7f67e47cbe83"). InnerVolumeSpecName "kube-api-access-cd5pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:16:52.670613 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.670574    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls" pod "390db20a-4526-419e-9304-7f67e47cbe83" (UID: "390db20a-4526-419e-9304-7f67e47cbe83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:52.671463 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.671406    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs" pod "390db20a-4526-419e-9304-7f67e47cbe83" (UID: "390db20a-4526-419e-9304-7f67e47cbe83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:16:52.672413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.672389    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/390db20a-4526-419e-9304-7f67e47cbe83-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:52.672513 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.672419    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:52.672513 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.672435    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5pr\" (UniqueName: \"kubernetes.io/projected/390db20a-4526-419e-9304-7f67e47cbe83-kube-api-access-cd5pr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:52.672513 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:52.672450    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/390db20a-4526-419e-9304-7f67e47cbe83-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:16:53.126258 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.126216    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:53.148790 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.148753    6931 generic.go:356] "Generic (PLEG): container finished" podID="390db20a-4526-419e-9304-7f67e47cbe83" containerID="d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7" exitCode=0
Oct 31 05:16:53.148962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.148804    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"
Oct 31 05:16:53.148962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.148835    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" event={"ID":"390db20a-4526-419e-9304-7f67e47cbe83","Type":"ContainerDied","Data":"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"}
Oct 31 05:16:53.148962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.148894    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm" event={"ID":"390db20a-4526-419e-9304-7f67e47cbe83","Type":"ContainerDied","Data":"177414ee3a9dc5d9daac7d82a2b8b286bcbf251e9d126d85cc027d16b1040b88"}
Oct 31 05:16:53.148962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.148918    6931 scope.go:122] "RemoveContainer" containerID="d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"
Oct 31 05:16:53.160759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.160733    6931 scope.go:122] "RemoveContainer" containerID="d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"
Oct 31 05:16:53.161333 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:16:53.161306    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7\": container with ID starting with d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7 not found: ID does not exist" containerID="d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"
Oct 31 05:16:53.161435 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.161343    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7"} err="failed to get container status \"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7\": rpc error: code = NotFound desc = could not find container \"d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7\": container with ID starting with d2c56d186b10de511239adaf829976f2993f5069fa203a640cf5de04a7bce2d7 not found: ID does not exist"
Oct 31 05:16:53.199472 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.199431    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 05:16:53.211106 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.211073    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-766728-7cc9cb67cb-m4hsm"]
Oct 31 05:16:53.916607 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:53.916573    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:54.121299 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.121263    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390db20a-4526-419e-9304-7f67e47cbe83" path="/var/lib/kubelet/pods/390db20a-4526-419e-9304-7f67e47cbe83/volumes"
Oct 31 05:16:54.125347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.125313    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:54.681305 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.681273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:54.993499 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.993401    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:54.999739 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.999704    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:16:54.999977 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:54.999947    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" containerID="cri-o://ddea88d368b21425b5db4ae0e51eb2bbdbac3e938b2dcd583f2f46f8e99d7861" gracePeriod=30
Oct 31 05:16:55.210001 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:55.209953    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:55.923059 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:55.923015    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:56.818746 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:56.818708    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:57.686734 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:57.686695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:57.893340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:57.893301    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:58.440078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:58.440043    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:58.685054 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:58.685010    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:58.902551 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:58.902513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:59.622993 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:59.622949    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:16:59.647823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:59.647791    6931 patch_prober.go:29] interesting pod/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 05:16:59.647968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:16:59.647845    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 05:17:00.189527 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.189489    6931 generic.go:356] "Generic (PLEG): container finished" podID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerID="ddea88d368b21425b5db4ae0e51eb2bbdbac3e938b2dcd583f2f46f8e99d7861" exitCode=0
Oct 31 05:17:00.189982 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.189545    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" event={"ID":"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3","Type":"ContainerDied","Data":"ddea88d368b21425b5db4ae0e51eb2bbdbac3e938b2dcd583f2f46f8e99d7861"}
Oct 31 05:17:00.266865 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.266800    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:17:00.351364 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351328    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351371    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351405    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351442    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351477    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351575 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351546    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.351873 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351832    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:17:00.351873 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.351826    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:00.352060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.352039    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:00.352145 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.352129    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.352259 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.352242    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.352447 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.352407    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:00.352662 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.352640    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\") pod \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\" (UID: \"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3\") "
Oct 31 05:17:00.353056 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.353024    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:00.353195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.353108    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.353195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.353150    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.353195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.353191    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.353396 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.353206    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.354049 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.354013    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "kube-api-access-q9h5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:00.354135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.354067    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:00.354135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.354128    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:00.354422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.354406    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo" pod "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" (UID: "1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:17:00.454486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.454454    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.454486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.454480    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.454486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.454491    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9h5d\" (UniqueName: \"kubernetes.io/projected/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-kube-api-access-q9h5d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.454715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.454502    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.454715 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.454511    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:00.609549 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:00.609507    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:01.195806 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.195776    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"
Oct 31 05:17:01.196277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.195801    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p" event={"ID":"1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3","Type":"ContainerDied","Data":"92d01fb2285e8cd93ed5c641780186c683bcc7ca23118412f3af3e280a20f20f"}
Oct 31 05:17:01.196277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.195865    6931 scope.go:122] "RemoveContainer" containerID="ddea88d368b21425b5db4ae0e51eb2bbdbac3e938b2dcd583f2f46f8e99d7861"
Oct 31 05:17:01.260653 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.260618    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:17:01.272081 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.272051    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-338f2e-openshift-default-67bb4686c-b4n6p"]
Oct 31 05:17:01.999612 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:01.999569    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:02.121755 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:02.121724    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" path="/var/lib/kubelet/pods/1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3/volumes"
Oct 31 05:17:02.203194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:02.203146    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:02.523952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:02.523908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:02.733817 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:02.733775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:04.399291 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:04.399245    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:04.615489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:04.615449    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:04.915293 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:04.915242    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:05.135172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:05.135130    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:06.267422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:06.267334    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:06.480829 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:06.480792    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:06.687694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:06.687658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:08.201578 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:08.201537    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:08.422981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:08.422944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:09.198912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.198880    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:09.411676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.411635    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:09.539064 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.539023    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 05:17:09.539279 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.539252    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" podUID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" containerName="maas-api" containerID="cri-o://d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4" gracePeriod=30
Oct 31 05:17:09.724838 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.724753    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:09.809902 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.809819    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 05:17:09.857941 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.857902    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") pod \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") "
Oct 31 05:17:09.858147 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.857990    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\" (UniqueName: \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\") pod \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") "
Oct 31 05:17:09.858147 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.858114    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\") pod \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") "
Oct 31 05:17:09.858310 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.858255    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\") pod \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\" (UID: \"74ccaa85-2ff7-4212-a176-bf6645a0eeae\") "
Oct 31 05:17:09.858945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.858919    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca" pod "74ccaa85-2ff7-4212-a176-bf6645a0eeae" (UID: "74ccaa85-2ff7-4212-a176-bf6645a0eeae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:17:09.861082 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.861057    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx" pod "74ccaa85-2ff7-4212-a176-bf6645a0eeae" (UID: "74ccaa85-2ff7-4212-a176-bf6645a0eeae"). InnerVolumeSpecName "kube-api-access-h5qlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:09.861499 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.861473    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs" pod "74ccaa85-2ff7-4212-a176-bf6645a0eeae" (UID: "74ccaa85-2ff7-4212-a176-bf6645a0eeae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:09.861670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.861650    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls" pod "74ccaa85-2ff7-4212-a176-bf6645a0eeae" (UID: "74ccaa85-2ff7-4212-a176-bf6645a0eeae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:09.959628 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.959592    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:09.959628 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.959625    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccaa85-2ff7-4212-a176-bf6645a0eeae-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:09.959628 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.959635    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccaa85-2ff7-4212-a176-bf6645a0eeae-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:09.959869 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:09.959646    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qlx\" (UniqueName: \"kubernetes.io/projected/74ccaa85-2ff7-4212-a176-bf6645a0eeae-kube-api-access-h5qlx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:10.255700 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.255659    6931 generic.go:356] "Generic (PLEG): container finished" podID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" containerID="d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4" exitCode=0
Oct 31 05:17:10.255909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.255694    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" event={"ID":"74ccaa85-2ff7-4212-a176-bf6645a0eeae","Type":"ContainerDied","Data":"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"}
Oct 31 05:17:10.255909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.255722    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"
Oct 31 05:17:10.255909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.255797    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v" event={"ID":"74ccaa85-2ff7-4212-a176-bf6645a0eeae","Type":"ContainerDied","Data":"d2c5efdfed3250c849a8ae9d3464ae20a5973c7838d6177f9951ac65989fa686"}
Oct 31 05:17:10.255909 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.255829    6931 scope.go:122] "RemoveContainer" containerID="d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"
Oct 31 05:17:10.268481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.268452    6931 scope.go:122] "RemoveContainer" containerID="d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"
Oct 31 05:17:10.268959 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:10.268930    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4\": container with ID starting with d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4 not found: ID does not exist" containerID="d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"
Oct 31 05:17:10.269068 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.268969    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4"} err="failed to get container status \"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4\": rpc error: code = NotFound desc = could not find container \"d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4\": container with ID starting with d5f43af7e332e5aada897ce21ff6bac63e7366f6f1d185b90206de92e80a99a4 not found: ID does not exist"
Oct 31 05:17:10.307492 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.307450    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 05:17:10.311172 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.311135    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b2a372-678c9776dc-2r29v"]
Oct 31 05:17:10.408096 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:10.408011    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:10.758306 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:10.758224    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:17:11.955690 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:11.955658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:12.121966 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:12.121921    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" path="/var/lib/kubelet/pods/74ccaa85-2ff7-4212-a176-bf6645a0eeae/volumes"
Oct 31 05:17:12.553418 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:12.553380    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:12.759149 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:12.759108    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:14.367720 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:14.367636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:14.599923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:14.599890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:15.153735 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:15.153695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:15.553960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:15.553925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:16.018420 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:16.018325    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:16.575725 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:16.575680    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:18.431340 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:18.431294    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:18.651149 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:18.651116    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:19.276305 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:19.276267    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:19.747712 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:19.747674    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:20.181241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:20.181180    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:20.406853 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:20.406815    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:20.868879 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:20.868842    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 05:17:21.096203 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:21.096142    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:21.866174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:21.866126    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:22.325385 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:22.325345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:24.147303 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:24.147264    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:25.118475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:25.118427    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 05:17:25.129961 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:25.129928    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 05:17:25.336112 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:25.336066    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:25.916098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:25.916058    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:26.123255 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:26.123212    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:27.779698 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:27.779664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:28.030563 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:28.030470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:28.984603 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:28.984566    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:29.718374 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:29.718330    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:31.509025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:31.508981    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:32.141637 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:32.141576    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:32.427274 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:32.427183    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:33.579264 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:33.579229    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:33.787359 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:33.787307    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:35.889521 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:35.889480    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main" containerID="cri-o://b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7" gracePeriod=45
Oct 31 05:17:35.940371 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:35.940335    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:36.155597 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.155530    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 05:17:36.240981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.240946    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\" (UniqueName: \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241176 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.240997    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241176 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241046    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241176 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241089    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241187    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241282    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\") pod \"581d4326-a691-421a-8f6b-3da791d651dd\" (UID: \"581d4326-a691-421a-8f6b-3da791d651dd\") "
Oct 31 05:17:36.241370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241289    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:36.241535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241429    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:36.241593 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241572    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:36.241706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241689    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.241769 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241710    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.241769 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.241724    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.243490 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.243460    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "kube-api-access-m2vhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:36.243694 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.243667    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:36.243764 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.243677    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm" pod "581d4326-a691-421a-8f6b-3da791d651dd" (UID: "581d4326-a691-421a-8f6b-3da791d651dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:36.342461 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.342425    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/581d4326-a691-421a-8f6b-3da791d651dd-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.342461 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.342456    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/581d4326-a691-421a-8f6b-3da791d651dd-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.342461 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.342465    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vhf\" (UniqueName: \"kubernetes.io/projected/581d4326-a691-421a-8f6b-3da791d651dd-kube-api-access-m2vhf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:36.396611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.396576    6931 generic.go:356] "Generic (PLEG): container finished" podID="581d4326-a691-421a-8f6b-3da791d651dd" containerID="b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7" exitCode=0
Oct 31 05:17:36.396814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.396620    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"
Oct 31 05:17:36.396814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.396665    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" event={"ID":"581d4326-a691-421a-8f6b-3da791d651dd","Type":"ContainerDied","Data":"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"}
Oct 31 05:17:36.396814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.396711    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k" event={"ID":"581d4326-a691-421a-8f6b-3da791d651dd","Type":"ContainerDied","Data":"df2aba72dfa25f2ba9940d071a8aefa74d20208221f8d0ae4a0ef9afd0fef5fc"}
Oct 31 05:17:36.396814 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.396733    6931 scope.go:122] "RemoveContainer" containerID="b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"
Oct 31 05:17:36.406890 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.406868    6931 scope.go:122] "RemoveContainer" containerID="b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"
Oct 31 05:17:36.407260 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:36.407226    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7\": container with ID starting with b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7 not found: ID does not exist" containerID="b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"
Oct 31 05:17:36.407406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.407273    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7"} err="failed to get container status \"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7\": rpc error: code = NotFound desc = could not find container \"b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7\": container with ID starting with b18dfee8d8ff5f03378867f27a18c3239425919a35695fc81552f8a39a0077e7 not found: ID does not exist"
Oct 31 05:17:36.446096 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.446065    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 05:17:36.449745 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:36.449717    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4c2387/distinct-2-w3-4c2387-kserve-5d8d54567d-kfx5k"]
Oct 31 05:17:38.121923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:38.121889    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581d4326-a691-421a-8f6b-3da791d651dd" path="/var/lib/kubelet/pods/581d4326-a691-421a-8f6b-3da791d651dd/volumes"
Oct 31 05:17:39.143585 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:39.143548    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:40.140365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.140326    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" containerID="cri-o://1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7" gracePeriod=45
Oct 31 05:17:40.152580 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.152544    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" containerID="cri-o://97ac8c4d17d3de2a4f1c2e4ad0ce993fef86491e55e07783b527ac27dfe4a25a" gracePeriod=45
Oct 31 05:17:40.207780 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.207737    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": dial tcp 10.134.0.58:8000: connect: connection refused"
Oct 31 05:17:40.258724 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.258689    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused"
Oct 31 05:17:40.397570 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.397508    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 05:17:40.419922 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.419893    6931 generic.go:356] "Generic (PLEG): container finished" podID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerID="97ac8c4d17d3de2a4f1c2e4ad0ce993fef86491e55e07783b527ac27dfe4a25a" exitCode=0
Oct 31 05:17:40.420038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.419934    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" event={"ID":"b94979e4-1d5a-473e-9e91-75af7a823ea5","Type":"ContainerDied","Data":"97ac8c4d17d3de2a4f1c2e4ad0ce993fef86491e55e07783b527ac27dfe4a25a"}
Oct 31 05:17:40.420038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.419987    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg" event={"ID":"b94979e4-1d5a-473e-9e91-75af7a823ea5","Type":"ContainerDied","Data":"30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2"}
Oct 31 05:17:40.420038 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.420003    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30b81a8fff46afd527ae630515b0dcf15eddb4b8e4d01bacb8298ae3d27c83f2"
Oct 31 05:17:40.421395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.421374    6931 generic.go:356] "Generic (PLEG): container finished" podID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerID="1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7" exitCode=0
Oct 31 05:17:40.421515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.421420    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"
Oct 31 05:17:40.421515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.421429    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" event={"ID":"b877bbfa-18c7-4a63-90db-81fd73c8c02e","Type":"ContainerDied","Data":"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"}
Oct 31 05:17:40.421515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.421457    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h" event={"ID":"b877bbfa-18c7-4a63-90db-81fd73c8c02e","Type":"ContainerDied","Data":"cc8347bad7186e76fe469c49fc7e99bd47b38930bd4b00402b11cbd2b6fd825d"}
Oct 31 05:17:40.421515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.421478    6931 scope.go:122] "RemoveContainer" containerID="1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"
Oct 31 05:17:40.430574 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.430556    6931 scope.go:122] "RemoveContainer" containerID="1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"
Oct 31 05:17:40.430842 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:40.430824    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7\": container with ID starting with 1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7 not found: ID does not exist" containerID="1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"
Oct 31 05:17:40.430915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.430858    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7"} err="failed to get container status \"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7\": rpc error: code = NotFound desc = could not find container \"1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7\": container with ID starting with 1744107ab1fab3c6da5c4d889a6b2207da3365740d3b5d7a7c929575decfe8f7 not found: ID does not exist"
Oct 31 05:17:40.431779 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.431761    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 05:17:40.485514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485483    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485541    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485575    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485599    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\" (UniqueName: \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485652    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485683    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\" (UniqueName: \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.485753 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485733    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.486185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485805    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.486185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485883    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\") pod \"b94979e4-1d5a-473e-9e91-75af7a823ea5\" (UID: \"b94979e4-1d5a-473e-9e91-75af7a823ea5\") "
Oct 31 05:17:40.486185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.485958    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.486185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.486002    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.486185 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.486038    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\") pod \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\" (UID: \"b877bbfa-18c7-4a63-90db-81fd73c8c02e\") "
Oct 31 05:17:40.487281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.486915    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.487281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.486999    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.487281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.487254    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.487528 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.487504    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.487528 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.487499    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.487731 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.487711    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.490589 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.490546    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.490589 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.490551    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:40.490780 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.490593    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "kube-api-access-jx225". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:40.490780 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.490629    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "kube-api-access-92t4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:40.491554 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.491529    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs" pod "b877bbfa-18c7-4a63-90db-81fd73c8c02e" (UID: "b877bbfa-18c7-4a63-90db-81fd73c8c02e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:40.491979 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.491966    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm" pod "b94979e4-1d5a-473e-9e91-75af7a823ea5" (UID: "b94979e4-1d5a-473e-9e91-75af7a823ea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:17:40.587317 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587282    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587317 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587315    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587329    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587341    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587352    6931 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b877bbfa-18c7-4a63-90db-81fd73c8c02e-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587362    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b877bbfa-18c7-4a63-90db-81fd73c8c02e-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587372    6931 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587383    6931 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b94979e4-1d5a-473e-9e91-75af7a823ea5-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587395    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx225\" (UniqueName: \"kubernetes.io/projected/b94979e4-1d5a-473e-9e91-75af7a823ea5-kube-api-access-jx225\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587407    6931 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587420    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92t4b\" (UniqueName: \"kubernetes.io/projected/b877bbfa-18c7-4a63-90db-81fd73c8c02e-kube-api-access-92t4b\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.587538 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.587431    6931 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b94979e4-1d5a-473e-9e91-75af7a823ea5-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:40.759442 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:40.759399    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:17:40.767795 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.767766    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 05:17:40.779481 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:40.779454    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/distinct-w0-965f61-kserve-696d7476-84n6h"]
Oct 31 05:17:41.427329 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:41.427251    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"
Oct 31 05:17:41.478353 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:41.478311    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 05:17:41.489915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:41.489880    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-965f61/premium-w0-965f61-kserve-687bd6b55c-wc7kg"]
Oct 31 05:17:42.121016 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:42.120977    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" path="/var/lib/kubelet/pods/b877bbfa-18c7-4a63-90db-81fd73c8c02e/volumes"
Oct 31 05:17:42.121429 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:42.121414    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" path="/var/lib/kubelet/pods/b94979e4-1d5a-473e-9e91-75af7a823ea5/volumes"
Oct 31 05:17:42.177737 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:42.177688    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:43.063169 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.063117    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 05:17:43.063645 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.063321    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" podUID="4138af25-08f1-468a-896a-bc1770415bdd" containerName="maas-api" containerID="cri-o://dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5" gracePeriod=30
Oct 31 05:17:43.237702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.237665    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:43.336220 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.336134    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 05:17:43.416119 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.416082    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\") pod \"4138af25-08f1-468a-896a-bc1770415bdd\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") "
Oct 31 05:17:43.416333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.416135    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\" (UniqueName: \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\") pod \"4138af25-08f1-468a-896a-bc1770415bdd\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") "
Oct 31 05:17:43.416333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.416206    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\") pod \"4138af25-08f1-468a-896a-bc1770415bdd\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") "
Oct 31 05:17:43.416333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.416323    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\") pod \"4138af25-08f1-468a-896a-bc1770415bdd\" (UID: \"4138af25-08f1-468a-896a-bc1770415bdd\") "
Oct 31 05:17:43.416589 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.416564    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca" pod "4138af25-08f1-468a-896a-bc1770415bdd" (UID: "4138af25-08f1-468a-896a-bc1770415bdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:17:43.418447 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.418423    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs" pod "4138af25-08f1-468a-896a-bc1770415bdd" (UID: "4138af25-08f1-468a-896a-bc1770415bdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:43.418447 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.418437    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls" pod "4138af25-08f1-468a-896a-bc1770415bdd" (UID: "4138af25-08f1-468a-896a-bc1770415bdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:43.418637 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.418428    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw" pod "4138af25-08f1-468a-896a-bc1770415bdd" (UID: "4138af25-08f1-468a-896a-bc1770415bdd"). InnerVolumeSpecName "kube-api-access-lbvbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:43.438747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.438714    6931 generic.go:356] "Generic (PLEG): container finished" podID="4138af25-08f1-468a-896a-bc1770415bdd" containerID="dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5" exitCode=0
Oct 31 05:17:43.438915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.438763    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"
Oct 31 05:17:43.438915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.438792    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" event={"ID":"4138af25-08f1-468a-896a-bc1770415bdd","Type":"ContainerDied","Data":"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"}
Oct 31 05:17:43.438915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.438848    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d" event={"ID":"4138af25-08f1-468a-896a-bc1770415bdd","Type":"ContainerDied","Data":"570bf2f2a602b23eac6e5f12ab031371e142e113a5e861be8823142c19c43496"}
Oct 31 05:17:43.438915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.438871    6931 scope.go:122] "RemoveContainer" containerID="dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"
Oct 31 05:17:43.448968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.448944    6931 scope.go:122] "RemoveContainer" containerID="dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"
Oct 31 05:17:43.449276 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:43.449257    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5\": container with ID starting with dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5 not found: ID does not exist" containerID="dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"
Oct 31 05:17:43.449354 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.449290    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5"} err="failed to get container status \"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5\": rpc error: code = NotFound desc = could not find container \"dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5\": container with ID starting with dabc311af8dc5638e03048c2799f40c07044df692ae7dcdfb6ab33cefe8ab5b5 not found: ID does not exist"
Oct 31 05:17:43.488705 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.488672    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 05:17:43.492834 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.492811    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4c2387-79759997df-fph2d"]
Oct 31 05:17:43.518118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.518076    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:43.518118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.518108    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138af25-08f1-468a-896a-bc1770415bdd-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:43.518118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.518119    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbvbw\" (UniqueName: \"kubernetes.io/projected/4138af25-08f1-468a-896a-bc1770415bdd-kube-api-access-lbvbw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:43.518118 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:43.518129    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4138af25-08f1-468a-896a-bc1770415bdd-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:44.121840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:44.121802    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4138af25-08f1-468a-896a-bc1770415bdd" path="/var/lib/kubelet/pods/4138af25-08f1-468a-896a-bc1770415bdd/volumes"
Oct 31 05:17:44.544798 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:44.544766    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:44.757059 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:44.757016    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:45.035055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.035025    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 05:17:45.035263 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.035234    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" podUID="165b9045-5aea-45a8-9092-f6559243a0e9" containerName="maas-api" containerID="cri-o://0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba" gracePeriod=30
Oct 31 05:17:45.166469 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.166432    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:45.304965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.304937    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 05:17:45.336673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.336628    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\") pod \"165b9045-5aea-45a8-9092-f6559243a0e9\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") "
Oct 31 05:17:45.336850 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.336679    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\") pod \"165b9045-5aea-45a8-9092-f6559243a0e9\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") "
Oct 31 05:17:45.336850 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.336739    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\" (UniqueName: \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\") pod \"165b9045-5aea-45a8-9092-f6559243a0e9\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") "
Oct 31 05:17:45.336850 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.336787    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\") pod \"165b9045-5aea-45a8-9092-f6559243a0e9\" (UID: \"165b9045-5aea-45a8-9092-f6559243a0e9\") "
Oct 31 05:17:45.337134 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.337104    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca" pod "165b9045-5aea-45a8-9092-f6559243a0e9" (UID: "165b9045-5aea-45a8-9092-f6559243a0e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:17:45.338884 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.338862    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls" pod "165b9045-5aea-45a8-9092-f6559243a0e9" (UID: "165b9045-5aea-45a8-9092-f6559243a0e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:45.338974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.338921    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv" pod "165b9045-5aea-45a8-9092-f6559243a0e9" (UID: "165b9045-5aea-45a8-9092-f6559243a0e9"). InnerVolumeSpecName "kube-api-access-hbcqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:17:45.338974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.338940    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs" pod "165b9045-5aea-45a8-9092-f6559243a0e9" (UID: "165b9045-5aea-45a8-9092-f6559243a0e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:17:45.373988 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.373947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:45.438577 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.438544    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:45.438747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.438581    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/165b9045-5aea-45a8-9092-f6559243a0e9-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:45.438747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.438598    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcqv\" (UniqueName: \"kubernetes.io/projected/165b9045-5aea-45a8-9092-f6559243a0e9-kube-api-access-hbcqv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:45.438747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.438614    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/165b9045-5aea-45a8-9092-f6559243a0e9-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:17:45.450929 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.450897    6931 generic.go:356] "Generic (PLEG): container finished" podID="165b9045-5aea-45a8-9092-f6559243a0e9" containerID="0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba" exitCode=0
Oct 31 05:17:45.451067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.450945    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"
Oct 31 05:17:45.451067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.450952    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" event={"ID":"165b9045-5aea-45a8-9092-f6559243a0e9","Type":"ContainerDied","Data":"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"}
Oct 31 05:17:45.451067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.450997    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8" event={"ID":"165b9045-5aea-45a8-9092-f6559243a0e9","Type":"ContainerDied","Data":"7d3c6c4684a9ebe5bd1651c448e5afbc47b4730850d44c3624257d0dbe47e902"}
Oct 31 05:17:45.451067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.451019    6931 scope.go:122] "RemoveContainer" containerID="0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"
Oct 31 05:17:45.459962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.459942    6931 scope.go:122] "RemoveContainer" containerID="0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"
Oct 31 05:17:45.460298 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:17:45.460272    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba\": container with ID starting with 0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba not found: ID does not exist" containerID="0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"
Oct 31 05:17:45.460382 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.460310    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba"} err="failed to get container status \"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba\": rpc error: code = NotFound desc = could not find container \"0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba\": container with ID starting with 0b56dcb85e833f1fab2c90481569714858f8546d64ac3df2e2b50397d66589ba not found: ID does not exist"
Oct 31 05:17:45.511285 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.511250    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 05:17:45.514947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:45.514909    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-965f61-85895dc44d-hb9l8"]
Oct 31 05:17:46.127400 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:46.127363    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165b9045-5aea-45a8-9092-f6559243a0e9" path="/var/lib/kubelet/pods/165b9045-5aea-45a8-9092-f6559243a0e9/volumes"
Oct 31 05:17:47.842364 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:47.842320    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:48.047029 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:48.046986    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:50.867967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:50.867923    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:51.089986 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:51.089951    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:57.944713 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:57.944676    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:58.161406 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:58.161359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:58.541650 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:58.541605    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:58.800294 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:58.800200    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:59.194738 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:59.194655    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:59.554515 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:59.554480    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:59.768692 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:59.768653    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:17:59.979837 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:17:59.979748    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:02.067456 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:02.067415    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:02.280974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:02.280928    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:03.756896 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:03.756859    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:04.689877 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:04.689833    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:04.896390 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:04.896355    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:04.973913 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:04.973799    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 05:18:04.974121 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:04.974071    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" containerID="cri-o://d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e" gracePeriod=30
Oct 31 05:18:05.117255 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:05.117215    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:05.816783 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:05.816747    6931 scope.go:122] "RemoveContainer" containerID="97ac8c4d17d3de2a4f1c2e4ad0ce993fef86491e55e07783b527ac27dfe4a25a"
Oct 31 05:18:05.945699 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:05.945662    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:06.155759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:06.155678    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:06.680649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:06.680618    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sqjb4_faeda3ac-ef2c-4bd5-a821-465f8bae2cd8/kuadrant-console-plugin/0.log"
Oct 31 05:18:08.377252 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:08.377205    6931 patch_prober.go:29] interesting pod/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 05:18:08.377642 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:08.377268    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 05:18:09.045751 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:09.045719    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:18:09.478201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:09.478103    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:18:09.804813 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:09.804778    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:10.023520 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.023477    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:10.217299 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.217269    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:18:10.229569 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.229542    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 05:18:10.298141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298102    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298147    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298200    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298305    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298370 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298366    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298427    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298484    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298488    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298512    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298541    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298615 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298605    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\") pod \"14645be7-c5dc-4dd0-8518-c29ff76916e4\" (UID: \"14645be7-c5dc-4dd0-8518-c29ff76916e4\") "
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298621    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298688    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298731    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298931    6931 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14645be7-c5dc-4dd0-8518-c29ff76916e4-istiod-ca-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298946    6931 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.298955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298955    6931 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-workload-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.299309 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298967    6931 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-data\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.299309 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.298984    6931 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-credential-socket\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.300758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.300734    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 05:18:10.300868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.300756    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:18:10.300868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.300773    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "kube-api-access-vm7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:18:10.300868 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.300772    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo" pod "14645be7-c5dc-4dd0-8518-c29ff76916e4" (UID: "14645be7-c5dc-4dd0-8518-c29ff76916e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 05:18:10.399676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.399590    6931 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.399676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.399625    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm7kx\" (UniqueName: \"kubernetes.io/projected/14645be7-c5dc-4dd0-8518-c29ff76916e4-kube-api-access-vm7kx\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.399676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.399636    6931 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-envoy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.399676 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.399645    6931 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14645be7-c5dc-4dd0-8518-c29ff76916e4-istio-podinfo\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:18:10.431901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.431868    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:18:10.581815 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.581777    6931 generic.go:356] "Generic (PLEG): container finished" podID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerID="d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e" exitCode=0
Oct 31 05:18:10.582262 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.581825    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"
Oct 31 05:18:10.582262 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.581823    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" event={"ID":"14645be7-c5dc-4dd0-8518-c29ff76916e4","Type":"ContainerDied","Data":"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"}
Oct 31 05:18:10.582262 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.581956    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr" event={"ID":"14645be7-c5dc-4dd0-8518-c29ff76916e4","Type":"ContainerDied","Data":"cb45975bb674c98d9a8e048f594a3a883c7f390843520d04db20d1781527b63b"}
Oct 31 05:18:10.582262 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.581986    6931 scope.go:122] "RemoveContainer" containerID="d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"
Oct 31 05:18:10.591273 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.591253    6931 scope.go:122] "RemoveContainer" containerID="d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"
Oct 31 05:18:10.591531 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:18:10.591512    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e\": container with ID starting with d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e not found: ID does not exist" containerID="d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"
Oct 31 05:18:10.591581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.591540    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e"} err="failed to get container status \"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e\": rpc error: code = NotFound desc = could not find container \"d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e\": container with ID starting with d455a713080bec29c81019a90586bb16d1acbf6e27c3f42a45561f3f3597268e not found: ID does not exist"
Oct 31 05:18:10.636983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.636946    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 05:18:10.648586 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:10.648558    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-965f61-openshift-default-55854964c-gr6vr"]
Oct 31 05:18:10.758641 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:18:10.758554    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:18:12.121055 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:12.121010    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" path="/var/lib/kubelet/pods/14645be7-c5dc-4dd0-8518-c29ff76916e4/volumes"
Oct 31 05:18:12.170115 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:12.170077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:12.389976 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:12.389890    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:15.245065 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:15.245018    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:15.736098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:15.736061    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:15.946438 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:15.946402    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:18.368626 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:18.368585    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:18.583529 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:18.583494    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:20.079881 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:20.079840    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:20.289732 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:20.289699    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:22.006455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:22.006421    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:22.833279 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:22.833244    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:23.042395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:23.042357    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:23.813782 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:23.813739    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:25.608128 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:25.608087    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:27.405788 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:27.405752    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:28.074365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:28.074331    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:28.985651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:28.985607    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:29.296135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:29.296101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:31.109225 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:31.109186    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:32.816672 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:32.816633    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:34.816109 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:34.816071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:36.606555 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:36.606523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:38.401665 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:38.401621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:40.202067 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:40.202025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:40.758253 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:18:40.758212    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:18:42.002573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:42.002531    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:43.722204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:43.722143    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:44.942598 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:44.942511    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:45.152371 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:45.152333    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:45.358901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:45.358865    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:47.837885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:47.837853    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:48.046006 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:48.045970    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:50.701061 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:50.701022    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:57.049747 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:57.049706    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:18:57.276253 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:18:57.276207    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:04.864706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:04.864664    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:10.758110 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:19:10.758065    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:19:11.120984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:11.120944    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:11.331433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:11.331397    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:27.546959 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:27.546920    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:29.572020 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:29.571986    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:34.046756 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:34.046718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:34.255976 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:19:34.255934    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:19:40.758218 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:19:40.758149    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:20:04.941488 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:20:04.941438    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:20:10.769659 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:20:10.769618    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:20:27.194634 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:20:27.194587    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:20:27.405098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:20:27.405064    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:20:40.757486 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:20:40.757446    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:21:01.768593 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:01.768543    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:21:02.125201 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:02.125139    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:21:03.740182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:03.740115    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:21:05.072230 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:05.072190    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:21:06.187304 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:06.187259    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:21:06.187956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:06.187921    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:21:10.758385 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:21:10.758340    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:21:21.474307 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:21.474270    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:21:40.757900 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:21:40.757856    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:21:45.209462 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:21:45.205077    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:05.224363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:05.224323    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:05.500716 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:05.500623    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:10.759368 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:22:10.759328    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:22:13.398391 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:13.398343    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:22:13.399077 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:13.399042    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:22:22.259350 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:22.259312    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:22.757708 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:22.757674    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:22.999229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:22.999177    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:25.952813 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:25.952775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:26.164533 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:26.164488    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:27.261097 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261059    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261529    6931 cpu_manager.go:414] "containerMap: removing container" podUID="390db20a-4526-419e-9304-7f67e47cbe83" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261544    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390db20a-4526-419e-9304-7f67e47cbe83" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261561    6931 cpu_manager.go:414] "containerMap: removing container" podUID="785982dc-fdb6-4553-b782-ffef892542b8" containerName="test-curl-7096-024b07"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261566    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785982dc-fdb6-4553-b782-ffef892542b8" containerName="test-curl-7096-024b07"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261577    6931 cpu_manager.go:414] "containerMap: removing container" podUID="4138af25-08f1-468a-896a-bc1770415bdd" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261583    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4138af25-08f1-468a-896a-bc1770415bdd" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261589    6931 cpu_manager.go:414] "containerMap: removing container" podUID="165b9045-5aea-45a8-9092-f6559243a0e9" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261593    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165b9045-5aea-45a8-9092-f6559243a0e9" containerName="maas-api"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261599    6931 cpu_manager.go:414] "containerMap: removing container" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy"
Oct 31 05:22:27.261604 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261604    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261614    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261618    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261631    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261638    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261651    6931 cpu_manager.go:414] "containerMap: removing container" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261659    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261672    6931 cpu_manager.go:414] "containerMap: removing container" podUID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261679    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261693    6931 cpu_manager.go:414] "containerMap: removing container" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261700    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261710    6931 cpu_manager.go:414] "containerMap: removing container" podUID="955d9376-e518-4754-8434-175eb794b910" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261717    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955d9376-e518-4754-8434-175eb794b910" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261726    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261733    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261747    6931 cpu_manager.go:414] "containerMap: removing container" podUID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261753    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261763    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261769    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261778    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261785    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261793    6931 cpu_manager.go:414] "containerMap: removing container" podUID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261800    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261810    6931 cpu_manager.go:414] "containerMap: removing container" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261817    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261824    6931 cpu_manager.go:414] "containerMap: removing container" podUID="bc0203d3-e103-4d9a-a98c-ff68b4606859" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261831    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0203d3-e103-4d9a-a98c-ff68b4606859" containerName="main"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261861    6931 cpu_manager.go:414] "containerMap: removing container" podUID="4c31e963-b71e-4edb-a266-0baf09315cd6" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261869    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c31e963-b71e-4edb-a266-0baf09315cd6" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261881    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261887    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261968    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261977    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c31e963-b71e-4edb-a266-0baf09315cd6" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261982    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261988    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="390db20a-4526-419e-9304-7f67e47cbe83" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.261996    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="785982dc-fdb6-4553-b782-ffef892542b8" containerName="test-curl-7096-024b07"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262002    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="4138af25-08f1-468a-896a-bc1770415bdd" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262007    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="165b9045-5aea-45a8-9092-f6559243a0e9" containerName="maas-api"
Oct 31 05:22:27.261996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262014    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb18b2d-5cdf-4f8d-9d9f-a8785b488ba3" containerName="istio-proxy"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262019    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262024    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b877bbfa-18c7-4a63-90db-81fd73c8c02e" containerName="main"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262030    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="581d4326-a691-421a-8f6b-3da791d651dd" containerName="main"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262036    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="425452e2-5ddf-4958-8eb1-d4fd1ebba711" containerName="main"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262043    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9cd816-5815-4568-9bca-fdceb1a56b69" containerName="cleanup"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262048    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="955d9376-e518-4754-8434-175eb794b910" containerName="maas-api"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262053    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262058    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262064    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccaa85-2ff7-4212-a176-bf6645a0eeae" containerName="maas-api"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262069    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262074    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94979e4-1d5a-473e-9e91-75af7a823ea5" containerName="main"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262079    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f640ad6-1426-42c4-8ed3-d5c6ef753214" containerName="main"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262085    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="14645be7-c5dc-4dd0-8518-c29ff76916e4" containerName="istio-proxy"
Oct 31 05:22:27.263360 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.262089    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0203d3-e103-4d9a-a98c-ff68b4606859" containerName="main"
Oct 31 05:22:27.265331 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.265315    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.294504 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.294468    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5534f0-serving-cert\""
Oct 31 05:22:27.294721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.294532    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\""
Oct 31 05:22:27.294721 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.294685    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 05:22:27.296727 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.296700    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:22:27.431741 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.431702    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.431931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.431747    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.431931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.431893    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnzs\" (UniqueName: \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.432010 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.431947    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.459426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.459390    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:27.532991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.532955    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnzs\" (UniqueName: \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.533219 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.533019    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.533219 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.533116    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.533219 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.533145    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.533419 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:22:27.533260    6931 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-serving-cert: secret "maas-api-e2e-worker-single-main-5534f0-serving-cert" not found
Oct 31 05:22:27.533419 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:22:27.533323    6931 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls podName:b3cf5e10-9a01-4ffb-8e3f-464169d09361 nodeName:}" failed. No retries permitted until 2026-10-31 05:22:28.033307054 +0000 UTC m=+4644.066306712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls") pod "maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" (UID: "b3cf5e10-9a01-4ffb-8e3f-464169d09361") : secret "maas-api-e2e-worker-single-main-5534f0-serving-cert" not found
Oct 31 05:22:27.533749 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.533727    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.535641 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.535617    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.558151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.558123    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnzs\" (UniqueName: \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:27.672138 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.672101    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:27.885876 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:27.885779    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:28.037903 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:28.037865    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:28.040460 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:28.040428    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:28.178253 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:28.178148    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:28.476088 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:28.475991    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:22:28.480108 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:22:28.480083    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cf5e10_9a01_4ffb_8e3f_464169d09361.slice/crio-4a59bf082b75819a60d54a6aecc46f6c99cb028f8df8b7d749d73c9861677d32 WatchSource:0}: Error finding container 4a59bf082b75819a60d54a6aecc46f6c99cb028f8df8b7d749d73c9861677d32: Status 404 returned error can't find the container with id 4a59bf082b75819a60d54a6aecc46f6c99cb028f8df8b7d749d73c9861677d32
Oct 31 05:22:28.481416 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:28.481401    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 05:22:29.042227 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:29.042191    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" event={"ID":"b3cf5e10-9a01-4ffb-8e3f-464169d09361","Type":"ContainerStarted","Data":"4a59bf082b75819a60d54a6aecc46f6c99cb028f8df8b7d749d73c9861677d32"}
Oct 31 05:22:29.185144 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:29.185109    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:30.047962 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:30.047923    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" event={"ID":"b3cf5e10-9a01-4ffb-8e3f-464169d09361","Type":"ContainerStarted","Data":"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"}
Oct 31 05:22:30.048389 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:30.048000    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:30.089977 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:30.089919    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" podStartSLOduration=2.529306516 podStartE2EDuration="3.089905594s" totalImagesPullingTime="560.599078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:22:30.088929137 +0000 UTC m=+4646.121928815" watchObservedRunningTime="2026-10-31 05:22:30.089905594 +0000 UTC m=+4646.122905273"
Oct 31 05:22:30.880125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:30.880070    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:32.581512 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:32.581472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:32.916455 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:32.916358    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:34.279436 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:34.279393    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:35.814766 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:35.814723    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:36.058526 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:36.058492    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:22:37.488791 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:37.488751    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:39.226063 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:39.226025    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:40.759392 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:22:40.759352    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:22:40.892803 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:40.892754    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:41.103704 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:41.103662    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:41.607954 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:41.607920    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:41.841875 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:41.841830    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:42.596675 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:42.596636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:43.263621 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:43.263581    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:44.378036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:44.378003    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:46.099438 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:46.099399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:47.766058 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:47.766021    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:49.387141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:49.387091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:50.975087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:50.975054    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:52.656002 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:52.655959    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:54.373797 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:54.373758    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:55.983434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:55.983391    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:57.585510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:57.585467    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:22:59.289821 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:22:59.289788    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:00.904223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:00.904182    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:02.596121 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:02.596074    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:04.281363 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:04.281320    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:05.415673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:05.415631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:05.901032 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:05.900997    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:07.492451 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:07.492404    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:09.187346 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:09.187312    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:10.758475 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:23:10.758432    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:23:10.887224 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:10.887183    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:12.483414 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:12.483377    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:14.181683 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:14.181602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:15.896015 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:15.895976    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:17.587315 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:17.587273    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:19.187376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:19.187334    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:20.804179 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:20.804120    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:22.531131 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:22.531086    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:24.186763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:24.186728    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:24.721217 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:24.721141    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:25.244775 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:25.244735    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:25.793759 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:25.793718    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:27.493645 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:27.493613    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:29.076758 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:29.076722    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:29.753658 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:29.753620    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:30.682947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:30.682907    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:31.489404 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:31.489365    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:31.838711 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:31.838677    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:32.046386 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:32.046324    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:32.472660 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:32.472621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:33.769440 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:33.769390    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:23:33.770052 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:33.769652    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" podUID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" containerName="maas-api" containerID="cri-o://9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94" gracePeriod=30
Oct 31 05:23:33.942505 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:33.942464    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:34.222502 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.222473    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:23:34.312913 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.312828    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\") pod \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") "
Oct 31 05:23:34.312913 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.312869    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\") pod \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") "
Oct 31 05:23:34.313111 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.313002    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\" (UniqueName: \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\") pod \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") "
Oct 31 05:23:34.313184 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.313114    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") pod \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\" (UID: \"b3cf5e10-9a01-4ffb-8e3f-464169d09361\") "
Oct 31 05:23:34.313270 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.313250    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca" pod "b3cf5e10-9a01-4ffb-8e3f-464169d09361" (UID: "b3cf5e10-9a01-4ffb-8e3f-464169d09361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:23:34.313489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.313466    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cf5e10-9a01-4ffb-8e3f-464169d09361-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:23:34.315077 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.315054    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs" pod "b3cf5e10-9a01-4ffb-8e3f-464169d09361" (UID: "b3cf5e10-9a01-4ffb-8e3f-464169d09361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:23:34.315223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.315057    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs" pod "b3cf5e10-9a01-4ffb-8e3f-464169d09361" (UID: "b3cf5e10-9a01-4ffb-8e3f-464169d09361"). InnerVolumeSpecName "kube-api-access-hlnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:23:34.315223 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.315081    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls" pod "b3cf5e10-9a01-4ffb-8e3f-464169d09361" (UID: "b3cf5e10-9a01-4ffb-8e3f-464169d09361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:23:34.362712 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.362673    6931 generic.go:356] "Generic (PLEG): container finished" podID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" containerID="9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94" exitCode=0
Oct 31 05:23:34.362885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.362719    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"
Oct 31 05:23:34.362885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.362739    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" event={"ID":"b3cf5e10-9a01-4ffb-8e3f-464169d09361","Type":"ContainerDied","Data":"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"}
Oct 31 05:23:34.362885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.362793    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd" event={"ID":"b3cf5e10-9a01-4ffb-8e3f-464169d09361","Type":"ContainerDied","Data":"4a59bf082b75819a60d54a6aecc46f6c99cb028f8df8b7d749d73c9861677d32"}
Oct 31 05:23:34.362885 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.362826    6931 scope.go:122] "RemoveContainer" containerID="9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"
Oct 31 05:23:34.372193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.372151    6931 scope.go:122] "RemoveContainer" containerID="9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"
Oct 31 05:23:34.372472 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:23:34.372455    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94\": container with ID starting with 9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94 not found: ID does not exist" containerID="9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"
Oct 31 05:23:34.372550 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.372478    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94"} err="failed to get container status \"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94\": rpc error: code = NotFound desc = could not find container \"9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94\": container with ID starting with 9652f25e9c3300f75d8c63467b2e360a528e0cc1df12a2a7fc8520af8b72cd94 not found: ID does not exist"
Oct 31 05:23:34.413597 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.413557    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:23:34.414125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.414090    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnzs\" (UniqueName: \"kubernetes.io/projected/b3cf5e10-9a01-4ffb-8e3f-464169d09361-kube-api-access-hlnzs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:23:34.414125 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.414122    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:23:34.414280 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.414133    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cf5e10-9a01-4ffb-8e3f-464169d09361-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:23:34.417671 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:34.417643    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5534f0-7dfb7cd94f-mczgd"]
Oct 31 05:23:36.123493 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:36.123458    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" path="/var/lib/kubelet/pods/b3cf5e10-9a01-4ffb-8e3f-464169d09361/volumes"
Oct 31 05:23:37.367947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:37.367910    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:37.576434 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:23:37.576398    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:23:40.758533 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:23:40.758492    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:24:03.309287 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:03.309246    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:04.538690 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:04.538638    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:04.751648 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:04.751619    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:05.485010 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:05.484971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:05.957710 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:05.957676    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:06.203649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:06.203562    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:06.481863 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:06.481777    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:07.812508 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:07.812472    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:08.508954 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:08.508908    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:09.491279 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:09.491235    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:10.759543 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:24:10.759503    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:24:11.181433 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:11.181399    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:12.901565 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:12.901523    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:14.148683 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.148631    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"]
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149035    6931 cpu_manager.go:414] "containerMap: removing container" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149046    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46f9fbf-5f62-4a23-8774-65abd361dcc2" containerName="cleanup"
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149067    6931 cpu_manager.go:414] "containerMap: removing container" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149073    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3194872f-4856-4b5c-a6c5-a38e58a0c2d7" containerName="cleanup"
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149090    6931 cpu_manager.go:414] "containerMap: removing container" podUID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" containerName="maas-api"
Oct 31 05:24:14.149114 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149095    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" containerName="maas-api"
Oct 31 05:24:14.149407 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.149182    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cf5e10-9a01-4ffb-8e3f-464169d09361" containerName="maas-api"
Oct 31 05:24:14.153411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.153365    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.180939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.180606    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\""
Oct 31 05:24:14.180939 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.180884    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 05:24:14.182273 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.182250    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"]
Oct 31 05:24:14.305699 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.305663    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.305931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.305729    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.305931 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.305906    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxs4\" (UniqueName: \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.407521 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.407434    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.407683 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.407525    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.407683 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.407590    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxs4\" (UniqueName: \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.408143 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.408119    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.409976 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.409948    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.435060 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.435034    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxs4\" (UniqueName: \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.469633 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.469595    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:14.500911 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.500878    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:14.787799 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.787764    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"]
Oct 31 05:24:14.791500 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:24:14.791471    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590b45d9_6d19_4a95_ae62_cc5340e8775b.slice/crio-f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c WatchSource:0}: Error finding container f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c: Status 404 returned error can't find the container with id f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c
Oct 31 05:24:14.811025 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:14.810993    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:15.011636 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:15.011602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:15.574396 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:15.574357    6931 generic.go:356] "Generic (PLEG): container finished" podID="590b45d9-6d19-4a95-ae62-cc5340e8775b" containerID="fd8aee68ce29d2692595a69adfef32fffb98dd431a0ac7339ee71f89fbb75893" exitCode=0
Oct 31 05:24:15.574760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:15.574401    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf" event={"ID":"590b45d9-6d19-4a95-ae62-cc5340e8775b","Type":"ContainerDied","Data":"fd8aee68ce29d2692595a69adfef32fffb98dd431a0ac7339ee71f89fbb75893"}
Oct 31 05:24:15.574760 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:15.574462    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf" event={"ID":"590b45d9-6d19-4a95-ae62-cc5340e8775b","Type":"ContainerStarted","Data":"f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c"}
Oct 31 05:24:16.171661 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.171621    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:16.714697 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.714673    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:16.833960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.833882    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\" (UniqueName: \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\") pod \"590b45d9-6d19-4a95-ae62-cc5340e8775b\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") "
Oct 31 05:24:16.833960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.833924    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\") pod \"590b45d9-6d19-4a95-ae62-cc5340e8775b\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") "
Oct 31 05:24:16.834151 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.833975    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\") pod \"590b45d9-6d19-4a95-ae62-cc5340e8775b\" (UID: \"590b45d9-6d19-4a95-ae62-cc5340e8775b\") "
Oct 31 05:24:16.834413 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.834387    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca" pod "590b45d9-6d19-4a95-ae62-cc5340e8775b" (UID: "590b45d9-6d19-4a95-ae62-cc5340e8775b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:24:16.835965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.835943    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca" pod "590b45d9-6d19-4a95-ae62-cc5340e8775b" (UID: "590b45d9-6d19-4a95-ae62-cc5340e8775b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:24:16.835965 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.835960    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4" pod "590b45d9-6d19-4a95-ae62-cc5340e8775b" (UID: "590b45d9-6d19-4a95-ae62-cc5340e8775b"). InnerVolumeSpecName "kube-api-access-xmxs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:24:16.935557 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.935519    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxs4\" (UniqueName: \"kubernetes.io/projected/590b45d9-6d19-4a95-ae62-cc5340e8775b-kube-api-access-xmxs4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:24:16.935557 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.935552    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-xks-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:24:16.935557 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:16.935561    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/590b45d9-6d19-4a95-ae62-cc5340e8775b-maas-api-openshift-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:24:17.584528 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:17.584486    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"
Oct 31 05:24:17.584528 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:17.584515    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf" event={"ID":"590b45d9-6d19-4a95-ae62-cc5340e8775b","Type":"ContainerDied","Data":"f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c"}
Oct 31 05:24:17.584800 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:17.584576    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f710b7e099569eccefa5c3e2cf39b2b4b500ae4155e2cc233045f8ea77f0a53c"
Oct 31 05:24:17.866457 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:17.866371    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:19.578333 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:19.578303    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:21.281452 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:21.281417    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:22.890119 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:22.890082    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:24.482014 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:24.481979    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:24.824213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:24.824138    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:25.037213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:25.037151    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:25.867141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:25.867103    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:26.387426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:26.387356    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:26.431936 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:26.431904    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"]
Oct 31 05:24:26.443810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:26.443777    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5534f0-hdbdf"]
Oct 31 05:24:26.600924 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:26.600885    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:26.909984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:26.909942    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:28.120430 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:28.120396    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590b45d9-6d19-4a95-ae62-cc5340e8775b" path="/var/lib/kubelet/pods/590b45d9-6d19-4a95-ae62-cc5340e8775b/volumes"
Oct 31 05:24:31.859522 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:31.859489    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:24:40.758915 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:24:40.758870    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:24:47.765629 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:24:47.765540    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:05.599981 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:05.599942    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:10.758310 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:25:10.758268    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:25:16.906714 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:16.906680    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:22.810573 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:22.810528    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:40.758949 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:25:40.758911    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:25:47.499754 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:47.499701    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:49.297915 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:49.297877    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:25:49.514519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:25:49.514486    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:26:05.721166 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:05.721124    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:26:10.759079 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:26:10.759038    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:26:40.758578 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:26:40.758542    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:26:45.996532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:45.996491    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:26:46.209540 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:46.209503    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:26:46.769702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:46.769656    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:26:51.183336 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:51.183282    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:26:51.183763 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:26:51.183357    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:27:05.833514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:27:05.833475    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:27:10.758794 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:27:10.758753    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:27:40.758200 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:27:40.758130    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:28:07.611681 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:07.611577    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:10.759840 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:28:10.759798    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:28:15.850350 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:15.850298    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:16.070355 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:16.070308    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:16.278755 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:16.278658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:29.977929 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:29.977893    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:30.201337 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:28:30.201296    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:28:40.758973 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:28:40.758934    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:29:06.114680 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:06.114636    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:29:09.031344 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:09.031302    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:29:10.757957 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:29:10.757915    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:29:40.758482 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:29:40.758443    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:29:42.685186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:42.684935    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:29:42.944832 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:42.944257    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:29:42.944832 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:42.944341    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:29:43.393374 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:43.393334    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:29:43.393594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:43.393424    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:29:57.939414 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:57.939365    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:29:57.939838 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:29:57.939443    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:30:01.543609 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.543566    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"]
Oct 31 05:30:01.544188 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.544170    6931 cpu_manager.go:414] "containerMap: removing container" podUID="590b45d9-6d19-4a95-ae62-cc5340e8775b" containerName="revoke-keys"
Oct 31 05:30:01.544240 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.544189    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590b45d9-6d19-4a95-ae62-cc5340e8775b" containerName="revoke-keys"
Oct 31 05:30:01.544296 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.544287    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="590b45d9-6d19-4a95-ae62-cc5340e8775b" containerName="revoke-keys"
Oct 31 05:30:01.547422 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.547405    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.564254 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.564231    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 05:30:01.576220 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.576194    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sfhzn\""
Oct 31 05:30:01.577622 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.577597    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"]
Oct 31 05:30:01.741282 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.741248    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8m64\" (UniqueName: \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.741282 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.741288    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.741512 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.741309    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.842199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.842087    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m64\" (UniqueName: \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.842199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.842133    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.842199 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.842152    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.843070 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.843043    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.844541 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.844517    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:01.865208 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:01.865182    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8m64\" (UniqueName: \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\") pod \"maas-api-revoke-sub-ae594979-lsbh2\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:02.160078 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:02.159993    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:02.457998 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:02.457879    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"]
Oct 31 05:30:02.515908 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:02.515871    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2" event={"ID":"33e83455-f8fb-4f90-adef-158a2c6587de","Type":"ContainerStarted","Data":"5b6760f1d26d854e98b29cf15a591847530366a9b0814ee064b512b67d11e0d2"}
Oct 31 05:30:03.523036 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:03.522992    6931 generic.go:356] "Generic (PLEG): container finished" podID="33e83455-f8fb-4f90-adef-158a2c6587de" containerID="97ff47c183b79fc9ad0169e8271b7ec597673b4a5f19401adec85df329bfd892" exitCode=0
Oct 31 05:30:03.523451 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:03.523078    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2" event={"ID":"33e83455-f8fb-4f90-adef-158a2c6587de","Type":"ContainerDied","Data":"97ff47c183b79fc9ad0169e8271b7ec597673b4a5f19401adec85df329bfd892"}
Oct 31 05:30:04.659398 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.659369    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:04.669399 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.669373    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\" (UniqueName: \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\") pod \"33e83455-f8fb-4f90-adef-158a2c6587de\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") "
Oct 31 05:30:04.669487 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.669464    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\") pod \"33e83455-f8fb-4f90-adef-158a2c6587de\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") "
Oct 31 05:30:04.669531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.669519    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\") pod \"33e83455-f8fb-4f90-adef-158a2c6587de\" (UID: \"33e83455-f8fb-4f90-adef-158a2c6587de\") "
Oct 31 05:30:04.669912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.669881    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca" pod "33e83455-f8fb-4f90-adef-158a2c6587de" (UID: "33e83455-f8fb-4f90-adef-158a2c6587de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:30:04.671486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.671466    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64" pod "33e83455-f8fb-4f90-adef-158a2c6587de" (UID: "33e83455-f8fb-4f90-adef-158a2c6587de"). InnerVolumeSpecName "kube-api-access-d8m64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:30:04.671569 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.671537    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca" pod "33e83455-f8fb-4f90-adef-158a2c6587de" (UID: "33e83455-f8fb-4f90-adef-158a2c6587de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:30:04.770956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.770918    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8m64\" (UniqueName: \"kubernetes.io/projected/33e83455-f8fb-4f90-adef-158a2c6587de-kube-api-access-d8m64\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:30:04.770956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.770950    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-xks-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:30:04.770956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:04.770960    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e83455-f8fb-4f90-adef-158a2c6587de-maas-api-openshift-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:30:05.534168 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:05.534115    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"
Oct 31 05:30:05.534168 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:05.534147    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2" event={"ID":"33e83455-f8fb-4f90-adef-158a2c6587de","Type":"ContainerDied","Data":"5b6760f1d26d854e98b29cf15a591847530366a9b0814ee064b512b67d11e0d2"}
Oct 31 05:30:05.534402 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:05.534206    6931 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b6760f1d26d854e98b29cf15a591847530366a9b0814ee064b512b67d11e0d2"
Oct 31 05:30:06.150424 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:06.150387    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:30:10.758590 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:30:10.758550    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:30:40.758892 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:30:40.758854    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:30:51.191409 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:51.191297    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:30:51.192327 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:30:51.192293    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:31:05.967347 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:31:05.967315    6931 scope.go:122] "RemoveContainer" containerID="fd8aee68ce29d2692595a69adfef32fffb98dd431a0ac7339ee71f89fbb75893"
Oct 31 05:31:06.380867 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:31:06.380830    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:31:10.758617 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:31:10.758577    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:31:13.484182 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:31:13.483219    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:31:40.815887 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:31:40.815840    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:32:06.490195 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:06.490134    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:10.758543 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:32:10.758496    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:32:27.425837 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:27.425799    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:27.911671 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:27.911628    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:28.122574 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:28.122536    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:31.932693 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:31.932650    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:32.145214 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:32.145144    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:33.441770 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:33.441726    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:33.648519 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:33.648481    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:33.857610 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:33.857578    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:35.183395 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:35.183359    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:36.880188 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:36.880119    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:38.608944 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:38.608889    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:40.303579 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:40.303542    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:40.759220 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:32:40.759116    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:32:42.009812 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:42.009761    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:43.679994 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:43.679955    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:45.289859 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:45.289817    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:46.854238 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:46.854204    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:47.094876 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:47.094835    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:47.494627 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:47.494585    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:47.710227 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:47.710176    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:48.940687 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:48.940648    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:50.215647 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:50.215613    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:50.714655 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:50.714602    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:52.433625 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:52.433582    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:52.742402 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:52.742305    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:54.100211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:54.100151    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:55.786991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:55.786951    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:57.372146 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:57.372107    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:32:58.978098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:32:58.978058    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:01.639974 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:01.639923    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:02.174774 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:02.174741    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:03.875702 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:03.875658    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:05.660268 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:05.660223    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:06.607321 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:06.607284    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:07.371799 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:07.371764    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:09.089550 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:09.089513    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:10.760582 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:33:10.760546    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:33:10.827318 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:10.827270    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:12.389595 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:12.389561    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:14.099518 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:14.099470    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:15.576826 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:15.576786    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:17.298130 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:17.298091    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:18.991286 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:18.991249    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:20.598607 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:20.598574    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:22.302697 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:22.302662    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:23.901818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:23.901775    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:25.601428 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:25.601386    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:27.174671 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:27.174629    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:28.879438 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:28.879400    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:30.479543 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:30.479504    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:32.131065 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:32.131012    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:32.472947 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:32.472858    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:32.955594 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:32.955552    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:33.774090 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:33.774033    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:35.499186 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:35.499121    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:37.076514 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:37.076476    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:37.498640 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:37.498527    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:38.153673 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:38.153632    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:38.368914 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:38.368871    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:38.823431 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:38.823382    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:39.476468 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:39.476428    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:40.223412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:40.223378    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:40.759875 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:33:40.759840    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:33:45.041775 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:45.041675    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:33:45.259989 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:33:45.259947    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:06.742532 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:06.742490    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:10.760283 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:34:10.760245    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:34:10.834531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:10.834493    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:11.057386 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:11.057345    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:12.255494 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:12.255427    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:12.475969 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:12.475925    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:12.688717 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:12.688667    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:14.016326 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:14.016291    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:14.301411 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:14.301363    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:15.188900 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:15.188860    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:15.704725 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:15.704681    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:17.470901 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:17.470861    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:19.175402 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:19.175369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:20.811241 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:20.811200    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:21.299956 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:21.299919    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:21.512668 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:21.512631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:22.487105 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:22.487071    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:24.079652 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:24.079611    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:25.681978 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:25.681937    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:27.377108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:27.377063    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:28.977412 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:28.977369    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:30.602742 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:30.602695    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:31.139381 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:31.139343    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:31.345706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:31.345645    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:32.048261 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:32.048217    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:32.583497 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:32.583453    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:32.799880 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:32.799836    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:33.129669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:33.129631    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:34:40.759728 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:34:40.759685    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:34:59.965784 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:34:59.965745    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:35:02.153680 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:35:02.153642    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:35:05.027581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:35:05.027544    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"]
Oct 31 05:35:05.040141 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:35:05.040093    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lsbh2"]
Oct 31 05:35:06.122284 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:35:06.122247    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e83455-f8fb-4f90-adef-158a2c6587de" path="/var/lib/kubelet/pods/33e83455-f8fb-4f90-adef-158a2c6587de/volumes"
Oct 31 05:35:06.864387 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:35:06.864354    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:35:10.758838 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:35:10.758798    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:35:40.760072 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:35:40.760032    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:36:06.018531 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:06.018494    6931 scope.go:122] "RemoveContainer" containerID="97ff47c183b79fc9ad0169e8271b7ec597673b4a5f19401adec85df329bfd892"
Oct 31 05:36:06.985255 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:06.985209    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:36:10.759100 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:36:10.759059    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:36:40.757970 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:36:40.757930    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:36:48.111623 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.111527    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:48.112281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.112078    6931 cpu_manager.go:414] "containerMap: removing container" podUID="33e83455-f8fb-4f90-adef-158a2c6587de" containerName="revoke-keys"
Oct 31 05:36:48.112281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.112093    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e83455-f8fb-4f90-adef-158a2c6587de" containerName="revoke-keys"
Oct 31 05:36:48.112281 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.112180    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e83455-f8fb-4f90-adef-158a2c6587de" containerName="revoke-keys"
Oct 31 05:36:48.115582 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.115553    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.133537 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.133505    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.133706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.133580    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.133706 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.133693    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.133834 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.133750    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2h2\" (UniqueName: \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.143221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.143183    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 05:36:48.143365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.143240    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qt6xg\""
Oct 31 05:36:48.143365 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.143248    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 05:36:48.147818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.147794    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:48.234952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.234913    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2h2\" (UniqueName: \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.235178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.234961    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.235178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.234995    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.235178 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.235024    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.235757 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.235733    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.237486 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.237466    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.237628 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.237605    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.257902 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.257869    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2h2\" (UniqueName: \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\") pod \"maas-api-b94596b94-66w7c\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") " pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.429388 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.429304    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:48.725383 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.725277    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:48.730063 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:36:48.730038    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db711b2_6c0c_4f54_8875_8bb75f2d0e51.slice/crio-9062f5dbd0ac5492ff5490310dab4e7c714a673f8ded00a9fe45110a28ceb7c5 WatchSource:0}: Error finding container 9062f5dbd0ac5492ff5490310dab4e7c714a673f8ded00a9fe45110a28ceb7c5: Status 404 returned error can't find the container with id 9062f5dbd0ac5492ff5490310dab4e7c714a673f8ded00a9fe45110a28ceb7c5
Oct 31 05:36:48.731441 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:48.731423    6931 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 05:36:49.671265 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:49.671228    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" event={"ID":"5db711b2-6c0c-4f54-8875-8bb75f2d0e51","Type":"ContainerStarted","Data":"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"}
Oct 31 05:36:49.671669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:49.671283    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" event={"ID":"5db711b2-6c0c-4f54-8875-8bb75f2d0e51","Type":"ContainerStarted","Data":"9062f5dbd0ac5492ff5490310dab4e7c714a673f8ded00a9fe45110a28ceb7c5"}
Oct 31 05:36:49.671669 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:49.671361    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:49.712535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:49.712481    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" podStartSLOduration=1.305501954 podStartE2EDuration="1.712468467s" totalImagesPullingTime="406.966513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:36:49.711271303 +0000 UTC m=+5505.744270982" watchObservedRunningTime="2026-10-31 05:36:49.712468467 +0000 UTC m=+5505.745468146"
Oct 31 05:36:55.664331 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.664294    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:55.664896 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.664514    6931 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" podUID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" containerName="maas-api" containerID="cri-o://a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43" gracePeriod=30
Oct 31 05:36:55.670073 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.670046    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:55.703923 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.703888    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"]
Oct 31 05:36:55.708209 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.708146    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.728013 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.727975    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"]
Oct 31 05:36:55.808870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.808831    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb29s\" (UniqueName: \"kubernetes.io/projected/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-kube-api-access-mb29s\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.809047 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.808877    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-metrics-certs\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.809047 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.808989    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-maas-api-tls\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.809047 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.809021    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-openshift-service-ca\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.910135 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.910102    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb29s\" (UniqueName: \"kubernetes.io/projected/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-kube-api-access-mb29s\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.910300 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.910150    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-metrics-certs\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.910300 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.910210    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-maas-api-tls\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.910300 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.910231    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-openshift-service-ca\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.911143 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.911035    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-openshift-service-ca\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.912839 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.912819    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-metrics-certs\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.913035 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.913009    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-maas-api-tls\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:55.930590 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.930533    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:55.933293 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:55.933264    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb29s\" (UniqueName: \"kubernetes.io/projected/9f4ffef7-9397-4c44-9ee8-367e479bbf8d-kube-api-access-mb29s\") pod \"maas-api-76f7dfc58b-g975g\" (UID: \"9f4ffef7-9397-4c44-9ee8-367e479bbf8d\") " pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:56.011426 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.011386    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\") pod \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") "
Oct 31 05:36:56.011611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.011473    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\") pod \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") "
Oct 31 05:36:56.011611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.011547    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\") pod \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") "
Oct 31 05:36:56.011611 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.011577    6931 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\" (UniqueName: \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\") pod \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\" (UID: \"5db711b2-6c0c-4f54-8875-8bb75f2d0e51\") "
Oct 31 05:36:56.011952 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.011923    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca" pod "5db711b2-6c0c-4f54-8875-8bb75f2d0e51" (UID: "5db711b2-6c0c-4f54-8875-8bb75f2d0e51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 05:36:56.013688 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.013662    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs" pod "5db711b2-6c0c-4f54-8875-8bb75f2d0e51" (UID: "5db711b2-6c0c-4f54-8875-8bb75f2d0e51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:36:56.013796 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.013690    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls" pod "5db711b2-6c0c-4f54-8875-8bb75f2d0e51" (UID: "5db711b2-6c0c-4f54-8875-8bb75f2d0e51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 05:36:56.013796 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.013718    6931 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2" pod "5db711b2-6c0c-4f54-8875-8bb75f2d0e51" (UID: "5db711b2-6c0c-4f54-8875-8bb75f2d0e51"). InnerVolumeSpecName "kube-api-access-qk2h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 05:36:56.072098 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.072051    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:56.112967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.112926    6931 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-metrics-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:36:56.112967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.112961    6931 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:36:56.112967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.112971    6931 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:36:56.112967 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.112980    6931 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2h2\" (UniqueName: \"kubernetes.io/projected/5db711b2-6c0c-4f54-8875-8bb75f2d0e51-kube-api-access-qk2h2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Oct 31 05:36:56.370463 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.370415    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"]
Oct 31 05:36:56.374508 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:36:56.374481    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ffef7_9397_4c44_9ee8_367e479bbf8d.slice/crio-85b4c27b3ba3bdcf6dbfeaf1e62bde77ee561e061be7ca670afc1749b4491cd1 WatchSource:0}: Error finding container 85b4c27b3ba3bdcf6dbfeaf1e62bde77ee561e061be7ca670afc1749b4491cd1: Status 404 returned error can't find the container with id 85b4c27b3ba3bdcf6dbfeaf1e62bde77ee561e061be7ca670afc1749b4491cd1
Oct 31 05:36:56.705774 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.705741    6931 generic.go:356] "Generic (PLEG): container finished" podID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" containerID="a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43" exitCode=0
Oct 31 05:36:56.706211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.705787    6931 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c"
Oct 31 05:36:56.706211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.705824    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" event={"ID":"5db711b2-6c0c-4f54-8875-8bb75f2d0e51","Type":"ContainerDied","Data":"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"}
Oct 31 05:36:56.706211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.705877    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b94596b94-66w7c" event={"ID":"5db711b2-6c0c-4f54-8875-8bb75f2d0e51","Type":"ContainerDied","Data":"9062f5dbd0ac5492ff5490310dab4e7c714a673f8ded00a9fe45110a28ceb7c5"}
Oct 31 05:36:56.706211 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.705905    6931 scope.go:122] "RemoveContainer" containerID="a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"
Oct 31 05:36:56.706976 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.706955    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g" event={"ID":"9f4ffef7-9397-4c44-9ee8-367e479bbf8d","Type":"ContainerStarted","Data":"85b4c27b3ba3bdcf6dbfeaf1e62bde77ee561e061be7ca670afc1749b4491cd1"}
Oct 31 05:36:56.717008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.716985    6931 scope.go:122] "RemoveContainer" containerID="a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"
Oct 31 05:36:56.717385 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:36:56.717353    6931 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43\": container with ID starting with a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43 not found: ID does not exist" containerID="a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"
Oct 31 05:36:56.717489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.717394    6931 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43"} err="failed to get container status \"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43\": rpc error: code = NotFound desc = could not find container \"a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43\": container with ID starting with a3a6510dfcc7712b057a166701ba2f3367680cb9002930de39e892322467bc43 not found: ID does not exist"
Oct 31 05:36:56.755586 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.755552    6931 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:56.768429 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:56.768398    6931 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-b94596b94-66w7c"]
Oct 31 05:36:57.713879 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:57.713839    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g" event={"ID":"9f4ffef7-9397-4c44-9ee8-367e479bbf8d","Type":"ContainerStarted","Data":"2bf862fbd439a80d3ad4f5b23d5b9e48a1164453ddc5567eba3184e2a618252d"}
Oct 31 05:36:57.714368 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:57.714018    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:36:57.754823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:57.754766    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g" podStartSLOduration=2.463079197 podStartE2EDuration="2.754753685s" totalImagesPullingTime="291.674488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 05:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:36:57.754559334 +0000 UTC m=+5513.787559016" watchObservedRunningTime="2026-10-31 05:36:57.754753685 +0000 UTC m=+5513.787753363"
Oct 31 05:36:58.121458 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:36:58.121424    6931 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" path="/var/lib/kubelet/pods/5db711b2-6c0c-4f54-8875-8bb75f2d0e51/volumes"
Oct 31 05:37:03.725816 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:03.725786    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76f7dfc58b-g975g"
Oct 31 05:37:07.111776 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:07.111736    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:37:09.268097 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:09.268059    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:37:10.759751 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:37:10.759713    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:37:27.253906 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:27.253867    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:37:27.460847 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:27.460804    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:37:31.391410 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:31.391371    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:37:36.183818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:36.183304    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:37:36.183818 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:37:36.183380    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:37:40.758448 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:37:40.758407    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:38:07.228174 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:38:07.228114    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:38:10.759173 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:38:10.759116    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:38:40.759243 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:38:40.759202    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:38:42.943237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:38:42.942272    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:38:42.943237 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:38:42.942366    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:39:06.184335 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:39:06.184282    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:39:06.184995 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:39:06.184965    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:39:07.466523 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:39:07.466488    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:39:10.758802 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:39:10.758761    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:39:15.303008 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:39:15.302971    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:39:40.758731 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:39:40.758686    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:39:50.182251 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:39:50.182137    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:40:07.594276 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:40:07.594234    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:40:10.834531 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:40:10.834448    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:40:21.181339 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:40:21.181290    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:40:21.181992 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:40:21.181955    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wzfdx" podUID="9e490550-c470-4717-9185-798d55ef73b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:40:40.821233 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:40:40.821192    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:40:54.280945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:40:54.280904    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:41:07.732805 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:07.732760    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:41:10.766255 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:41:10.766206    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:41:12.942803 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:12.942274    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:41:12.942803 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:12.942365    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:41:40.758782 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:41:40.758739    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:41:42.941438 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:42.941312    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5znqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:41:42.942062 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:42.942028    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5znqb" podUID="a7b9e7bd-eab8-43ed-a754-ed016aeab9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:41:43.382330 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:43.382276    6931 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 05:41:43.382602 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:41:43.382487    6931 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-jd8wd" podUID="de27e8de-c83f-48d2-b90f-c1bad562a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 05:42:03.604229 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:03.604194    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:04.094376 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:04.094335    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:04.617221 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:04.617187    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:04.811026 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:04.810990    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:42:06.251696 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:06.251664    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f7dfc58b-g975g_9f4ffef7-9397-4c44-9ee8-367e479bbf8d/maas-api/0.log"
Oct 31 05:42:07.836968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:07.836927    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:42:10.268955 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:10.268924    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fwgt5_2b8dc5b1-daac-4b97-a08f-c104a612e45a/postgres/0.log"
Oct 31 05:42:10.758624 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:42:10.758589    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:42:12.959738 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:12.959705    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sqjb4_faeda3ac-ef2c-4bd5-a821-465f8bae2cd8/kuadrant-console-plugin/0.log"
Oct 31 05:42:15.202830 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:15.202795    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-892tn_25f8bf9a-4406-4cc3-986f-35c546130809/kube-auth-proxy/0.log"
Oct 31 05:42:15.599311 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:15.599277    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5znqb_a7b9e7bd-eab8-43ed-a754-ed016aeab9c7/istio-proxy/0.log"
Oct 31 05:42:16.063033 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:16.063000    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-jd8wd_de27e8de-c83f-48d2-b90f-c1bad562a1d0/istio-proxy/0.log"
Oct 31 05:42:17.599510 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:17.599475    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-wzfdx_9e490550-c470-4717-9185-798d55ef73b2/istio-proxy/0.log"
Oct 31 05:42:32.400568 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:32.400525    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:42:32.737649 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:32.737566    6931 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log" limit=1024
Oct 31 05:42:36.806467 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:36.806434    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s4d8g_4502cc34-9b28-4f7e-a13b-594221372bbf/global-pull-secret-syncer/0.log"
Oct 31 05:42:37.568334 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:37.568294    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gk4cs_95ce4d22-fc05-4428-a9dd-651a3b1c4686/konnectivity-agent/0.log"
Oct 31 05:42:38.219467 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:38.219434    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-110.ec2.internal_50080b7deadb8e30dee000b7f0bbadfa/haproxy/0.log"
Oct 31 05:42:40.758790 ip-10-0-140-110 kubenswrapper[6931]: E1031 05:42:40.758753    6931 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jphhc" containerName="extractor"
Oct 31 05:42:41.689828 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:41.689790    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:41.806968 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:41.806935    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:43.489023 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:43.488983    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:43.637476 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:43.637444    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:43.767480 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:43.767383    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:43.928085 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:43.928049    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:44.484607 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:44.484523    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sqjb4_faeda3ac-ef2c-4bd5-a821-465f8bae2cd8/kuadrant-console-plugin/0.log"
Oct 31 05:42:46.468087 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.468052    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/alertmanager/0.log"
Oct 31 05:42:46.587372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.587328    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/config-reloader/0.log"
Oct 31 05:42:46.706415 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.706377    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/kube-rbac-proxy-web/0.log"
Oct 31 05:42:46.706610 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.706508    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:42:46.826259 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.826229    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/kube-rbac-proxy/0.log"
Oct 31 05:42:46.944668 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.944637    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/kube-rbac-proxy-metric/0.log"
Oct 31 05:42:46.966204 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:46.966146    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:42:47.061018 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.060983    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/prom-label-proxy/0.log"
Oct 31 05:42:47.178991 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.178905    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31fdcc1c-2f57-4ded-bbe7-1d07aea75705/init-config-reloader/0.log"
Oct 31 05:42:47.405489 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.405450    6931 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"]
Oct 31 05:42:47.405987 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.405972    6931 cpu_manager.go:414] "containerMap: removing container" podUID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" containerName="maas-api"
Oct 31 05:42:47.406034 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.405989    6931 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" containerName="maas-api"
Oct 31 05:42:47.406102 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.406091    6931 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db711b2-6c0c-4f54-8875-8bb75f2d0e51" containerName="maas-api"
Oct 31 05:42:47.409196 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.409177    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.429348 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.429271    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"]
Oct 31 05:42:47.452646 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.452612    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjhf4_de8cf693-3022-4ad6-a50a-82466ad009ba/kube-state-metrics/0.log"
Oct 31 05:42:47.461666 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.461633    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-podres\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.461870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.461689    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-lib-modules\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.461870 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.461832    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-sys\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.461984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.461874    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-proc\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.461984 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.461955    6931 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhf9d\" (UniqueName: \"kubernetes.io/projected/2c70c199-f613-46be-8f1b-0d0eeb387fc8-kube-api-access-jhf9d\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.521348 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.521302    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"openshift-service-ca.crt\""
Oct 31 05:42:47.521348 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.521329    6931 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n76t8\"/\"default-dockercfg-f2kxb\""
Oct 31 05:42:47.521348 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.521311    6931 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n76t8\"/\"kube-root-ca.crt\""
Oct 31 05:42:47.563131 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563082    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhf9d\" (UniqueName: \"kubernetes.io/projected/2c70c199-f613-46be-8f1b-0d0eeb387fc8-kube-api-access-jhf9d\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563145    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-podres\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563201    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-lib-modules\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563261    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-sys\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563298    6931 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-proc\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563349 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563326    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-podres\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563584 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563399    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-sys\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563584 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563411    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-lib-modules\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.563584 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.563418    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c70c199-f613-46be-8f1b-0d0eeb387fc8-proc\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.568942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.568915    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjhf4_de8cf693-3022-4ad6-a50a-82466ad009ba/kube-rbac-proxy-main/0.log"
Oct 31 05:42:47.585908 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.585878    6931 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhf9d\" (UniqueName: \"kubernetes.io/projected/2c70c199-f613-46be-8f1b-0d0eeb387fc8-kube-api-access-jhf9d\") pod \"perf-node-gather-daemonset-wxljn\" (UID: \"2c70c199-f613-46be-8f1b-0d0eeb387fc8\") " pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.685823 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.685737    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjhf4_de8cf693-3022-4ad6-a50a-82466ad009ba/kube-rbac-proxy-self/0.log"
Oct 31 05:42:47.722277 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.722245    6931 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:47.813108 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.813073    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b9ddc4b6b-k2l56_eb6c8de0-72bd-40d8-aa86-34805cf518d0/metrics-server/0.log"
Oct 31 05:42:47.931767 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:47.931728    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-brlvf_1fbb956a-7849-497a-9463-f85599ef4d57/monitoring-plugin/0.log"
Oct 31 05:42:48.017996 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.017902    6931 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"]
Oct 31 05:42:48.023374 ip-10-0-140-110 kubenswrapper[6931]: W1031 05:42:48.023336    6931 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2c70c199_f613_46be_8f1b_0d0eeb387fc8.slice/crio-5ed1a2007ffb3c1afe1e32ed45c51a07ee197377a6ecd50688fa1a130176a334 WatchSource:0}: Error finding container 5ed1a2007ffb3c1afe1e32ed45c51a07ee197377a6ecd50688fa1a130176a334: Status 404 returned error can't find the container with id 5ed1a2007ffb3c1afe1e32ed45c51a07ee197377a6ecd50688fa1a130176a334
Oct 31 05:42:48.251999 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.251963    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:48.431535 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.431499    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfnsk_f60643a3-3f4e-4308-a52e-0893b0fce7eb/node-exporter/0.log"
Oct 31 05:42:48.549945 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.549902    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfnsk_f60643a3-3f4e-4308-a52e-0893b0fce7eb/kube-rbac-proxy/0.log"
Oct 31 05:42:48.614942 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.614907    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:48.672650 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.672608    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfnsk_f60643a3-3f4e-4308-a52e-0893b0fce7eb/init-textfile/0.log"
Oct 31 05:42:48.991911 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.991866    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn" event={"ID":"2c70c199-f613-46be-8f1b-0d0eeb387fc8","Type":"ContainerStarted","Data":"a9548ec7754dbb4d471af8933776453c912407b8bcb00fa4df15e8c6ddf28873"}
Oct 31 05:42:48.992101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.991969    6931 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn" event={"ID":"2c70c199-f613-46be-8f1b-0d0eeb387fc8","Type":"ContainerStarted","Data":"5ed1a2007ffb3c1afe1e32ed45c51a07ee197377a6ecd50688fa1a130176a334"}
Oct 31 05:42:48.992101 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:48.992047    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:49.032469 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.032397    6931 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn" podStartSLOduration=2.0323816199999998 podStartE2EDuration="2.03238162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 05:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 05:42:49.031582102 +0000 UTC m=+5865.064581783" watchObservedRunningTime="2026-10-31 05:42:49.03238162 +0000 UTC m=+5865.065381302"
Oct 31 05:42:49.161269 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.161231    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nd47d_a345e1d5-f60a-473a-9cef-11db043e79df/kube-rbac-proxy-main/0.log"
Oct 31 05:42:49.281958 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.281894    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nd47d_a345e1d5-f60a-473a-9cef-11db043e79df/kube-rbac-proxy-self/0.log"
Oct 31 05:42:49.403810 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.403771    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nd47d_a345e1d5-f60a-473a-9cef-11db043e79df/openshift-state-metrics/0.log"
Oct 31 05:42:49.543846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.543761    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/prometheus/0.log"
Oct 31 05:42:49.661302 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.661268    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/config-reloader/0.log"
Oct 31 05:42:49.780213 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.780141    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/thanos-sidecar/0.log"
Oct 31 05:42:49.900708 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:49.900626    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/kube-rbac-proxy-web/0.log"
Oct 31 05:42:50.003563 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.003528    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:50.019327 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.019282    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/kube-rbac-proxy/0.log"
Oct 31 05:42:50.139294 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.139261    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/kube-rbac-proxy-thanos/0.log"
Oct 31 05:42:50.203912 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.203831    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:50.258193 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.258145    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b71d59e-2ca7-49a1-b07b-fb44ad47dbc3/init-config-reloader/0.log"
Oct 31 05:42:50.406257 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.406218    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:50.544372 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.544342    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:50.757961 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.757928    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8d6659cd-4n7qm_f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/telemeter-client/0.log"
Oct 31 05:42:50.876552 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.876472    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8d6659cd-4n7qm_f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/reload/0.log"
Oct 31 05:42:50.995651 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:50.995622    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b8d6659cd-4n7qm_f6f8fddd-031d-48c8-a22c-4b5c0bf69c5a/kube-rbac-proxy/0.log"
Oct 31 05:42:51.121410 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.121373    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/thanos-query/0.log"
Oct 31 05:42:51.240846 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.240733    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/kube-rbac-proxy-web/0.log"
Oct 31 05:42:51.358969 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.358934    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/kube-rbac-proxy/0.log"
Oct 31 05:42:51.478855 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.478804    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/prom-label-proxy/0.log"
Oct 31 05:42:51.598236 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.598199    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/kube-rbac-proxy-rules/0.log"
Oct 31 05:42:51.718475 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:51.718436    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68c888f8cf-rcbzw_35f518cc-282f-4dc6-9279-4e162eba47ab/kube-rbac-proxy-metrics/0.log"
Oct 31 05:42:53.359207 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:53.359146    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:42:53.648926 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:53.648809    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:42:55.008933 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:55.008905    6931 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n76t8/perf-node-gather-daemonset-wxljn"
Oct 31 05:42:55.045886 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:55.045840    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:55.387794 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:55.387758    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qdpw6_0517decd-0685-403b-b09f-0f7334b1b426/manager/0.log"
Oct 31 05:42:56.811296 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:56.811259    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:57.168894 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:57.168809    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8b55h_9d4dbbfa-848a-439f-bef4-172b3ea50d73/manager/0.log"
Oct 31 05:42:57.378762 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:57.378724    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:57.573855 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:57.573822    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9mrkv_176be731-5be3-437f-a60f-ec36fc3de64a/server/0.log"
Oct 31 05:42:59.059052 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:59.059005    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6nhz_b5d7df10-6133-426c-a8e1-48c78cdc1f44/dns/0.log"
Oct 31 05:42:59.173983 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:59.173948    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v6nhz_b5d7df10-6133-426c-a8e1-48c78cdc1f44/kube-rbac-proxy/0.log"
Oct 31 05:42:59.778840 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:42:59.778809    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-phzfq_acf269a0-e3d7-4932-ba07-f4f48ee0c787/dns-node-resolver/0.log"
Oct 31 05:43:00.564313 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:00.564272    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:43:00.766876 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:00.766832    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5qwzx_8cc7b10a-1f4d-4b8a-85b6-37bf95208d0d/manager/0.log"
Oct 31 05:43:01.396787 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:01.396754    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvz9v_ca7768dc-fc1a-405d-92cc-75e17eaccf19/node-ca/0.log"
Oct 31 05:43:01.494755 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:01.494716    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-892tn_25f8bf9a-4406-4cc3-986f-35c546130809/kube-auth-proxy/0.log"
Oct 31 05:43:01.640191 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:01.640143    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5znqb_a7b9e7bd-eab8-43ed-a754-ed016aeab9c7/istio-proxy/0.log"
Oct 31 05:43:01.768194 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:01.768097    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-jd8wd_de27e8de-c83f-48d2-b90f-c1bad562a1d0/istio-proxy/0.log"
Oct 31 05:43:02.232356 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:02.232297    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-wzfdx_9e490550-c470-4717-9185-798d55ef73b2/istio-proxy/0.log"
Oct 31 05:43:04.131960 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.131928    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-892tn_25f8bf9a-4406-4cc3-986f-35c546130809/kube-auth-proxy/0.log"
Oct 31 05:43:04.264692 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.264658    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-892tn_25f8bf9a-4406-4cc3-986f-35c546130809/kube-auth-proxy/0.log"
Oct 31 05:43:04.270820 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.270788    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5znqb_a7b9e7bd-eab8-43ed-a754-ed016aeab9c7/istio-proxy/0.log"
Oct 31 05:43:04.401658 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.401565    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5znqb_a7b9e7bd-eab8-43ed-a754-ed016aeab9c7/istio-proxy/0.log"
Oct 31 05:43:04.402448 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.402423    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-jd8wd_de27e8de-c83f-48d2-b90f-c1bad562a1d0/istio-proxy/0.log"
Oct 31 05:43:04.530456 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.530424    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-jd8wd_de27e8de-c83f-48d2-b90f-c1bad562a1d0/istio-proxy/0.log"
Oct 31 05:43:04.876099 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.876067    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-wzfdx_9e490550-c470-4717-9185-798d55ef73b2/istio-proxy/0.log"
Oct 31 05:43:04.982670 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:04.982636    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-wzfdx_9e490550-c470-4717-9185-798d55ef73b2/istio-proxy/0.log"
Oct 31 05:43:05.353666 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:05.353633    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58666cb9df-892tn_25f8bf9a-4406-4cc3-986f-35c546130809/kube-auth-proxy/0.log"
Oct 31 05:43:05.610134 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:05.610046    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5znqb_a7b9e7bd-eab8-43ed-a754-ed016aeab9c7/istio-proxy/0.log"
Oct 31 05:43:05.855581 ip-10-0-140-110 kubenswrapper[6931]: I1031 05:43:05.855548    6931 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-jd8wd_de27e8de-c83f-48d2-b90f-c1bad562a1d0/istio-proxy/0.log"